[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:07 dist-test-slave-2rbl krb5kdc[21645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:07 dist-test-slave-2rbl krb5kdc[21645](info): set up 2 sockets
Sep 29 10:23:07 dist-test-slave-2rbl krb5kdc[21645](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 10:23:08.902243 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:08 dist-test-slave-2rbl krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:09.028302 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:43339
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:43339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:09.139283 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:09.139559 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:09.139599 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:09.143667 21659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:09.143728 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:09.143747 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:09.143764 21659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:09.143827 21659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:09.149164 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:43339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:43339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:09.150308 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:09.151393 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:09.160362 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:09.160466 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:09.160534 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:09.160466 21659 server_base.cc:1034] running on GCE node
I20260929 10:23:09.161294 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:09.162442 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:09.163605 21659 hybrid_clock.cc:648] HybridClock initialized: now 1790677389163584 us; error 32 us; skew 500 ppm
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:09.167490 21659 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:09.168696 21659 webserver.cc:460] Webserver started at http://127.21.33.190:36791/ using document root <none> and password file <none>
I20260929 10:23:09.169392 21659 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:09.169489 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:09.169757 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:09.172698 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "bd7a3e48ecdc4186a4d54f6121a5010c"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "3c8f349d12167439bdf4f1bfa074d4b1"
server_key_iv: "ea03efb15fe53e3876761f1822610ba6"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.173422 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "bd7a3e48ecdc4186a4d54f6121a5010c"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "3c8f349d12167439bdf4f1bfa074d4b1"
server_key_iv: "ea03efb15fe53e3876761f1822610ba6"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.177188 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260929 10:23:09.179540 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.180737 21659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:09.180835 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "bd7a3e48ecdc4186a4d54f6121a5010c"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "3c8f349d12167439bdf4f1bfa074d4b1"
server_key_iv: "ea03efb15fe53e3876761f1822610ba6"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.180917 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:09.201292 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:09.204026 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:09.204241 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:09.211033 21659 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:43339
I20260929 10:23:09.211058 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:43339 every 8 connection(s)
I20260929 10:23:09.211988 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:09.214768 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 21659
I20260929 10:23:09.214823 21727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:09.214895 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:09.216185 21638 external_mini_cluster.cc:1428] Setting key 16a51eb7383c5e1397dedb958a5efe9b
I20260929 10:23:09.221220 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c: Bootstrap starting.
I20260929 10:23:09.223438 21727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:09.224207 21727 log.cc:824] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:09.226193 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c: No bootstrap required, opened a new log
I20260929 10:23:09.229635 21727 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43339 } }
I20260929 10:23:09.229912 21727 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:09.229971 21727 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7a3e48ecdc4186a4d54f6121a5010c, State: Initialized, Role: FOLLOWER
I20260929 10:23:09.230404 21727 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43339 } }
I20260929 10:23:09.230564 21727 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:09.230652 21727 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:09.230782 21727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:09.231901 21727 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43339 } }
I20260929 10:23:09.232255 21727 leader_election.cc:302] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7a3e48ecdc4186a4d54f6121a5010c; no voters:
I20260929 10:23:09.232599 21727 leader_election.cc:288] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:09.232678 21732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:09.232884 21732 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [term 1 LEADER]: Becoming Leader. State: Replica: bd7a3e48ecdc4186a4d54f6121a5010c, State: Running, Role: LEADER
I20260929 10:23:09.233219 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43339 } }
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:09.233952 21727 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:09.234469 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7a3e48ecdc4186a4d54f6121a5010c. Latest consensus state: current_term: 1 leader_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43339 } } }
I20260929 10:23:09.234583 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:09.235262 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7a3e48ecdc4186a4d54f6121a5010c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43339 } } }
I20260929 10:23:09.235376 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:09.236227 21736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:09.240442 21736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:09.241155 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.219347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.21.33.190:43339)
0929 10:23:09.219788 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:09.219803 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:09.220883 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 10:23:09.224070 (+ 3187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:09.224085 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:09.224454 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:09.224778 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:09.227064 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.228033 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.229271 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.229734 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.229918 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.234592 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:09.234600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:09.234615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:09.234653 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:09.237247 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.237924 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.237928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.237933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.238043 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.238466 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.238468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.238469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.238996 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:09.239199 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:09.239899 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 10:23:09.240195 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":147,"threads_started":1}
I20260929 10:23:09.246105 21736 catalog_manager.cc:1347] Generated new cluster ID: 871220d6312b414ea1a4413649c930ff
I20260929 10:23:09.246187 21736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:09.264412 21736 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:09.265785 21736 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:09.276697 21736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd7a3e48ecdc4186a4d54f6121a5010c: Generated new TSK 0
I20260929 10:23:09.277462 21736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:09.312395 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:43339
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:09.428256 21753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:09.428531 21753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:09.428576 21753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:09.432602 21753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:09.432668 21753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:09.432773 21753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:09.438483 21753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:43339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:09.439528 21753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:09.440583 21753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:09.449960 21758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:09.450040 21753 server_base.cc:1034] running on GCE node
W20260929 10:23:09.449968 21761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:09.449980 21759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:09.450624 21753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:09.451186 21753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:09.452376 21753 hybrid_clock.cc:648] HybridClock initialized: now 1790677389452365 us; error 31 us; skew 500 ppm
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:09.458632 21753 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:09.459538 21753 webserver.cc:460] Webserver started at http://127.21.33.129:36863/ using document root <none> and password file <none>
I20260929 10:23:09.460114 21753 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:09.460206 21753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:09.460438 21753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:09.462494 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "7a9a9f8d751547ca927587a529fa2f7e"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "218ce09a755e730305bbc167f600f9cc"
server_key_iv: "2c33e348a3f51d9f84d746aa96ce7a26"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.463053 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "7a9a9f8d751547ca927587a529fa2f7e"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "218ce09a755e730305bbc167f600f9cc"
server_key_iv: "2c33e348a3f51d9f84d746aa96ce7a26"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.466629 21753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:09.468933 21768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.470160 21753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:09.470297 21753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "7a9a9f8d751547ca927587a529fa2f7e"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "218ce09a755e730305bbc167f600f9cc"
server_key_iv: "2c33e348a3f51d9f84d746aa96ce7a26"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.470408 21753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:09.484465 21753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:09.487135 21753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:09.487329 21753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:09.487869 21753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:09.488649 21753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:09.488719 21753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.488797 21753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:09.488850 21753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.498175 21753 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:42561
I20260929 10:23:09.498278 21883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:42561 every 8 connection(s)
I20260929 10:23:09.499274 21753 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:09.509444 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 21753
I20260929 10:23:09.509531 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:09.509842 21638 external_mini_cluster.cc:1428] Setting key 0ba6cab05f7459292f91eb4ddc2ad3e6
I20260929 10:23:09.513439 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.502105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:59817 (local address 127.21.33.190:43339)
0929 10:23:09.502233 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:09.502236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:09.503253 (+ 1017us) server_negotiation.cc:407] Connection header received
0929 10:23:09.504068 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:09.504070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:09.504115 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:09.504187 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:09.505326 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.505857 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.506917 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.507137 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.507210 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.509743 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:09.509748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:09.509750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:09.509780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:09.511334 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.511812 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.511819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.511823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.511910 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.512225 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.512231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.512236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.512479 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:09.512583 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:09.513109 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 10:23:09.513258 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:09.514202 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.502477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:43339 (local address 127.21.33.129:59817)
0929 10:23:09.503113 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:09.503153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:09.503897 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:09.504344 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:09.504350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:09.504686 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:09.505152 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:09.505164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.505965 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.505967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:09.506815 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:09.506823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.507259 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.507262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:09.507393 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.507884 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:09.507899 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:09.509517 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:09.511443 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:09.511448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:09.511458 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:09.511710 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:09.512001 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:09.512003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:09.512005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:09.512134 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:09.512606 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:09.512611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:09.512862 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 10:23:09.513051 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 10:23:09.513401 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":111,"threads_started":1}
I20260929 10:23:09.515441 21884 heartbeater.cc:347] Connected to a master server at 127.21.33.190:43339
I20260929 10:23:09.515693 21884 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:09.516300 21884 heartbeater.cc:511] Master 127.21.33.190:43339 requested a full tablet report, sending...
I20260929 10:23:09.517843 21691 ts_manager.cc:194] Registered new tserver with Master: 7a9a9f8d751547ca927587a529fa2f7e (127.21.33.129:42561)
I20260929 10:23:09.518977 21691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:59817
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:09.538358 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:43339
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:09.649945 21889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:09.650210 21889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:09.650254 21889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:09.654284 21889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:09.654332 21889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:09.654392 21889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:09.659400 21889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:43339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:09.660346 21889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:09.661336 21889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:09.670646 21894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:09.670667 21895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:09.670688 21897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:09.671097 21889 server_base.cc:1034] running on GCE node
I20260929 10:23:09.671481 21889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:09.672075 21889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:09.673290 21889 hybrid_clock.cc:648] HybridClock initialized: now 1790677389673265 us; error 30 us; skew 500 ppm
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:09.676088 21889 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:09.676926 21889 webserver.cc:460] Webserver started at http://127.21.33.130:41867/ using document root <none> and password file <none>
I20260929 10:23:09.677584 21889 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:09.677649 21889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:09.677806 21889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:09.679745 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "2d76ba33793d4434af8deab4a33a2616"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "c6aeaffa3d6256c325bec25ec364057d"
server_key_iv: "abd084a4adf19f85d8ec4abbf1592558"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.680243 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "2d76ba33793d4434af8deab4a33a2616"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "c6aeaffa3d6256c325bec25ec364057d"
server_key_iv: "abd084a4adf19f85d8ec4abbf1592558"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.683724 21889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:23:09.686437 21904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.687578 21889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:09.687706 21889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "2d76ba33793d4434af8deab4a33a2616"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "c6aeaffa3d6256c325bec25ec364057d"
server_key_iv: "abd084a4adf19f85d8ec4abbf1592558"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.687811 21889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:09.702342 21889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:09.705312 21889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:09.705498 21889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:09.706054 21889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:09.706853 21889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:09.706923 21889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.706993 21889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:09.707046 21889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.715962 21889 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:42895
I20260929 10:23:09.715987 22018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:42895 every 8 connection(s)
I20260929 10:23:09.716878 21889 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:09.724848 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 21889
I20260929 10:23:09.724984 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:09.725358 21638 external_mini_cluster.cc:1428] Setting key ec8485d017487ce90f94e874e94e2f57
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:09.730290 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.718788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:45615 (local address 127.21.33.190:43339)
0929 10:23:09.718923 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:09.718926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:09.719924 (+ 998us) server_negotiation.cc:407] Connection header received
0929 10:23:09.720766 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:09.720769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:09.720815 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:09.720887 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:09.722094 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.722615 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.723684 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.723889 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.723962 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.726537 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:09.726571 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:09.726575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:09.726632 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:09.728173 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.728695 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.728701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.728703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.728802 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.729119 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.729124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.729128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.729406 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:09.729513 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:09.729982 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 10:23:09.730105 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:23:09.731156 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.719149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:43339 (local address 127.21.33.130:45615)
0929 10:23:09.719801 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:09.719841 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:09.720590 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:09.721048 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:09.721054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:09.721481 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:09.721972 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:09.721985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.722723 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.722726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:09.723574 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:09.723592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.724012 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.724014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:09.724151 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.724699 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:09.724715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:09.726388 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:09.728303 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:09.728312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:09.728329 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:09.728596 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:09.728895 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:09.728897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:09.728899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:09.729026 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:09.729512 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:09.729517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:09.729735 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 10:23:09.729932 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 10:23:09.730240 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":115,"threads_started":1}
I20260929 10:23:09.732333 22019 heartbeater.cc:347] Connected to a master server at 127.21.33.190:43339
I20260929 10:23:09.732587 22019 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:09.733088 22019 heartbeater.cc:511] Master 127.21.33.190:43339 requested a full tablet report, sending...
I20260929 10:23:09.734216 21691 ts_manager.cc:194] Registered new tserver with Master: 2d76ba33793d4434af8deab4a33a2616 (127.21.33.130:42895)
I20260929 10:23:09.734843 21691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:45615
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:09.752779 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:43339
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:09.863461 22024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:09.863716 22024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:09.863759 22024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:09.867996 22024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:09.868048 22024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:09.868106 22024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:09.873183 22024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:43339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:09.874248 22024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:09.875252 22024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:09.885015 22032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:09.885027 22029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:09.885042 22030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:09.885325 22024 server_base.cc:1034] running on GCE node
I20260929 10:23:09.885743 22024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:09.886343 22024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:09.887513 22024 hybrid_clock.cc:648] HybridClock initialized: now 1790677389887479 us; error 44 us; skew 500 ppm
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:09.890398 22024 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:09.891301 22024 webserver.cc:460] Webserver started at http://127.21.33.131:44679/ using document root <none> and password file <none>
I20260929 10:23:09.891911 22024 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:09.891999 22024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:09.892204 22024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:09.894248 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "49fdd6239159477989db74808abb0070"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "23b7f41853c1b4e14c9a365768bbeff5"
server_key_iv: "e411775e08e6a8c0d43e488d96cf729f"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.894801 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "49fdd6239159477989db74808abb0070"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "23b7f41853c1b4e14c9a365768bbeff5"
server_key_iv: "e411775e08e6a8c0d43e488d96cf729f"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.898492 22024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:09.900959 22039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.902204 22024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:09.902339 22024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "49fdd6239159477989db74808abb0070"
format_stamp: "Formatted at 2026-09-29 10:23:09 on dist-test-slave-2rbl"
server_key: "23b7f41853c1b4e14c9a365768bbeff5"
server_key_iv: "e411775e08e6a8c0d43e488d96cf729f"
server_key_version: "encryptionkey@0"
I20260929 10:23:09.902447 22024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:09.914773 22024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:09.917486 22024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:09.917752 22024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:09.918560 22024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:09.919445 22024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:09.919523 22024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.919593 22024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:09.919639 22024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:09.929273 22024 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:34215
I20260929 10:23:09.929301 22152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:34215 every 8 connection(s)
I20260929 10:23:09.930167 22024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:09.938966 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22024
I20260929 10:23:09.939214 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:09.939525 21638 external_mini_cluster.cc:1428] Setting key 099dde3279eb9ecb66b01c7d4291c5df
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:09.944116 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.931976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:45749 (local address 127.21.33.190:43339)
0929 10:23:09.932162 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:09.932167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:09.933192 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 10:23:09.934180 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:09.934183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:09.934232 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:09.934313 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:09.935593 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.936223 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.937384 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.937624 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.937701 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.940401 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:09.940404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:09.940406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:09.940430 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:09.942229 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.942830 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.942832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.942834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.942909 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.943220 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.943222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.943223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.943391 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:09.943465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:09.943902 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 10:23:09.943999 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 10:23:09.944911 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.932379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:43339 (local address 127.21.33.131:45749)
0929 10:23:09.933032 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:09.933071 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:09.933923 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:09.934473 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:09.934480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:09.934838 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:09.935440 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:09.935453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.936334 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.936337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:09.937246 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:09.937259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.937749 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.937751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:09.937902 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.938461 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:09.938476 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:09.940228 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:09.942388 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:09.942395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:09.942410 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:09.942737 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:09.942991 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:09.942993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:09.942995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:09.943126 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:09.943477 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:09.943482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:09.943704 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 10:23:09.943857 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 10:23:09.944099 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":123,"threads_started":1}
I20260929 10:23:09.945994 22153 heartbeater.cc:347] Connected to a master server at 127.21.33.190:43339
I20260929 10:23:09.946220 22153 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:09.946699 22153 heartbeater.cc:511] Master 127.21.33.190:43339 requested a full tablet report, sending...
I20260929 10:23:09.947700 21691 ts_manager.cc:194] Registered new tserver with Master: 49fdd6239159477989db74808abb0070 (127.21.33.131:34215)
I20260929 10:23:09.948273 21691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:45749
I20260929 10:23:09.954180 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:09.979836 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:09.970429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.21.33.129:42561)
0929 10:23:09.970700 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:09.970718 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:09.970747 (+ 29us) server_negotiation.cc:407] Connection header received
0929 10:23:09.970945 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:09.970952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:09.971143 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:09.971306 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:09.971980 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.972747 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.973643 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:09.974090 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:09.974214 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:09.976567 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:09.976576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:09.976589 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:09.976613 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:09.978176 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.978584 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.978589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.978590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.978667 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:09.978895 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:09.978897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:09.978899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:09.979126 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:09.979274 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:09.979463 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 10:23:09.979637 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
W20260929 10:23:09.980809 21796 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57188
Sep 29 10:23:09 dist-test-slave-2rbl krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:23:10 dist-test-slave-2rbl krb5kdc[21645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:10.009661 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:10.000964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.21.33.129:42561)
0929 10:23:10.001118 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:10.001123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:10.001280 (+ 157us) server_negotiation.cc:407] Connection header received
0929 10:23:10.001440 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:10.001444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:10.001498 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:10.001582 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:10.002273 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:10.002800 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:10.003654 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:10.003910 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:10.004020 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:10.006353 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:10.006356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:10.006358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:10.006379 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:10.007805 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:10.008276 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:10.008282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:10.008284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:10.008394 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:10.008762 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:10.008765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:10.008767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:10.008992 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:10.009127 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:10.009433 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 10:23:10.009539 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 10:23:10.010658 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 21753
I20260929 10:23:10.016484 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 21889
I20260929 10:23:10.023674 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22024
I20260929 10:23:10.028860 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 21659
2026-09-29T10:23:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2188 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:10 dist-test-slave-2rbl krb5kdc[22167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:10 dist-test-slave-2rbl krb5kdc[22167](info): set up 2 sockets
Sep 29 10:23:10 dist-test-slave-2rbl krb5kdc[22167](info): commencing operation
krb5kdc: starting...
W20260929 10:23:11.132673 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 10:23:11.451694 21638 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.319s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 10:23:11.658068 21638 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.206s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 10:23:11.985575 21638 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.327s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/test-user.keytab.
W20260929 10:23:12.935531 21638 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.950s user 0.001s sys 0.001s
Sep 29 10:23:12 dist-test-slave-2rbl krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
W20260929 10:23:13.999915 21638 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.33.190: real 1.040s user 0.003s sys 0.000s
I20260929 10:23:14.000521 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:36119
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:36119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:14.117034 22180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:14.117331 22180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:14.117372 22180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:14.121381 22180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:14.121444 22180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:14.121464 22180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:14.121480 22180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:14.121496 22180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:14.126823 22180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:36119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:36119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:14.127871 22180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:14.128890 22180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:14.137614 22185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:14.137642 22188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:14.137791 22186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:14.138187 22180 server_base.cc:1034] running on GCE node
I20260929 10:23:14.138844 22180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:14.140080 22180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:14.141301 22180 hybrid_clock.cc:648] HybridClock initialized: now 1790677394141265 us; error 49 us; skew 500 ppm
Sep 29 10:23:14 dist-test-slave-2rbl krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:14.144351 22180 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:14.145328 22180 webserver.cc:460] Webserver started at http://127.21.33.190:43485/ using document root <none> and password file <none>
I20260929 10:23:14.145897 22180 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:14.145959 22180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:14.146131 22180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:14.148370 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "82a3aeae798f49b984f19de7740f0232"
format_stamp: "Formatted at 2026-09-29 10:23:14 on dist-test-slave-2rbl"
server_key: "b914b4e464ec7ebc9f770772bf1a2438"
server_key_iv: "768c3766b09d2392415d76bfb5e59c2e"
server_key_version: "encryptionkey@0"
I20260929 10:23:14.148964 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "82a3aeae798f49b984f19de7740f0232"
format_stamp: "Formatted at 2026-09-29 10:23:14 on dist-test-slave-2rbl"
server_key: "b914b4e464ec7ebc9f770772bf1a2438"
server_key_iv: "768c3766b09d2392415d76bfb5e59c2e"
server_key_version: "encryptionkey@0"
I20260929 10:23:14.153100 22180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:23:14.155803 22195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:14.157109 22180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:14.157282 22180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "82a3aeae798f49b984f19de7740f0232"
format_stamp: "Formatted at 2026-09-29 10:23:14 on dist-test-slave-2rbl"
server_key: "b914b4e464ec7ebc9f770772bf1a2438"
server_key_iv: "768c3766b09d2392415d76bfb5e59c2e"
server_key_version: "encryptionkey@0"
I20260929 10:23:14.157384 22180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:14.183591 22180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:14.186856 22180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:14.187014 22180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:14.194850 22180 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:36119
I20260929 10:23:14.194880 22247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:36119 every 8 connection(s)
I20260929 10:23:14.195927 22180 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:14.197808 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22180
I20260929 10:23:14.197933 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:14.198262 21638 external_mini_cluster.cc:1428] Setting key 933e9ece4ec65496b55d2d5895300e12
I20260929 10:23:14.199369 22248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:14.206204 22248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232: Bootstrap starting.
Sep 29 10:23:14 dist-test-slave-2rbl krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:14.208818 22248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:14.209744 22248 log.cc:824] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:14.211758 22248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232: No bootstrap required, opened a new log
I20260929 10:23:14.213804 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:14.199757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.21.33.190:36119)
0929 10:23:14.200160 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:14.200182 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:14.200229 (+ 47us) server_negotiation.cc:407] Connection header received
0929 10:23:14.200882 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:14.200899 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:14.201315 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:14.201600 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:14.202545 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:14.203626 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:14.204865 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:14.205404 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:14.205525 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:14.208432 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:14.208437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:14.208451 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:14.208477 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:14.210555 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:14.211010 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:14.211016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:14.211022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:14.211149 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:14.211451 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:14.211455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:14.211457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:14.211875 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:14.212050 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:14.212416 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 10:23:14.212727 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":154,"threads_started":1}
I20260929 10:23:14.215246 22248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a3aeae798f49b984f19de7740f0232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36119 } }
I20260929 10:23:14.215507 22248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:14.215611 22248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a3aeae798f49b984f19de7740f0232, State: Initialized, Role: FOLLOWER
I20260929 10:23:14.216127 22248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a3aeae798f49b984f19de7740f0232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36119 } }
I20260929 10:23:14.216296 22248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:14.216391 22248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:14.216506 22248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:14.257422 22248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a3aeae798f49b984f19de7740f0232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36119 } }
I20260929 10:23:14.258018 22248 leader_election.cc:302] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82a3aeae798f49b984f19de7740f0232; no voters:
I20260929 10:23:14.258493 22248 leader_election.cc:288] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:14.258706 22253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:14.258965 22253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [term 1 LEADER]: Becoming Leader. State: Replica: 82a3aeae798f49b984f19de7740f0232, State: Running, Role: LEADER
I20260929 10:23:14.259344 22253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a3aeae798f49b984f19de7740f0232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36119 } }
I20260929 10:23:14.260102 22248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:14.261346 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82a3aeae798f49b984f19de7740f0232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a3aeae798f49b984f19de7740f0232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36119 } } }
I20260929 10:23:14.261720 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:14.261181 22255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82a3aeae798f49b984f19de7740f0232. Latest consensus state: current_term: 1 leader_uuid: "82a3aeae798f49b984f19de7740f0232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a3aeae798f49b984f19de7740f0232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36119 } } }
I20260929 10:23:14.262152 22255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:14.262252 22262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:14.265496 22262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:14.271459 22262 catalog_manager.cc:1347] Generated new cluster ID: e5a14c4945dc481eb0e7dfba4b3b80b0
I20260929 10:23:14.271540 22262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:14.288776 22262 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:14.289839 22262 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:14.299647 22262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82a3aeae798f49b984f19de7740f0232: Generated new TSK 0
I20260929 10:23:14.300403 22262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
W20260929 10:23:15.332598 21638 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.33.129: real 1.026s user 0.004s sys 0.000s
I20260929 10:23:15.333325 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36119
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:15.447209 22274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:15.447548 22274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:15.447649 22274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:15.451951 22274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:15.452035 22274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:15.452126 22274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:15.457500 22274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:15.458678 22274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:15.459723 22274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:15.470052 22279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:15.470077 22280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:15.470068 22282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:15.470337 22274 server_base.cc:1034] running on GCE node
I20260929 10:23:15.470821 22274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:15.471491 22274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:15.472724 22274 hybrid_clock.cc:648] HybridClock initialized: now 1790677395472692 us; error 51 us; skew 500 ppm
Sep 29 10:23:15 dist-test-slave-2rbl krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677395, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:15.475580 22274 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:15.476504 22274 webserver.cc:460] Webserver started at http://127.21.33.129:42233/ using document root <none> and password file <none>
I20260929 10:23:15.477083 22274 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:15.477174 22274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:15.477415 22274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:15.479480 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "0346ef7182394e2ba794f32ec9d2bc63"
format_stamp: "Formatted at 2026-09-29 10:23:15 on dist-test-slave-2rbl"
server_key: "77acec11dd165c6f1e326f5cc402dedf"
server_key_iv: "e986552efe3c352f16b53b350b862388"
server_key_version: "encryptionkey@0"
I20260929 10:23:15.480055 22274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "0346ef7182394e2ba794f32ec9d2bc63"
format_stamp: "Formatted at 2026-09-29 10:23:15 on dist-test-slave-2rbl"
server_key: "77acec11dd165c6f1e326f5cc402dedf"
server_key_iv: "e986552efe3c352f16b53b350b862388"
server_key_version: "encryptionkey@0"
I20260929 10:23:15.483876 22274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 10:23:15.486351 22289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:15.487610 22274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:15.487771 22274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "0346ef7182394e2ba794f32ec9d2bc63"
format_stamp: "Formatted at 2026-09-29 10:23:15 on dist-test-slave-2rbl"
server_key: "77acec11dd165c6f1e326f5cc402dedf"
server_key_iv: "e986552efe3c352f16b53b350b862388"
server_key_version: "encryptionkey@0"
I20260929 10:23:15.487886 22274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:15.512470 22274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:15.516040 22274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:15.516276 22274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:15.516917 22274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:15.517849 22274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:15.517922 22274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:15.517992 22274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:15.518039 22274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:15.527730 22274 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:42509
I20260929 10:23:15.527747 22403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:42509 every 8 connection(s)
I20260929 10:23:15.528626 22274 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:15.530118 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22274
I20260929 10:23:15.530206 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:15.530509 21638 external_mini_cluster.cc:1428] Setting key 5d86c63bf73c764534184576ee28f4f5
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Sep 29 10:23:15 dist-test-slave-2rbl krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677395, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:15.782467 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:15.530431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:57955 (local address 127.21.33.190:36119)
0929 10:23:15.530765 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:15.530769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:15.532285 (+ 1516us) server_negotiation.cc:407] Connection header received
0929 10:23:15.533685 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:15.533688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:15.533749 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:15.533844 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:15.535566 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:15.536326 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:15.537642 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:15.537906 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:15.538023 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:15.776992 (+238969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:15.777000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:15.777007 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:15.777051 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:15.779554 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:15.780407 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:15.780412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:15.780413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:15.780509 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:15.781016 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:15.781019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:15.781020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:15.781320 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:15.781406 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:15.782211 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 10:23:15.782328 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":67,"threads_started":1}
I20260929 10:23:15.783766 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:15.530944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36119 (local address 127.21.33.129:57955)
0929 10:23:15.532138 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:15.532191 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:15.533472 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:15.534179 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:15.534192 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:15.534744 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:15.535418 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:15.535434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:15.536453 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:15.536459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:15.537509 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:15.537518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:15.538069 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:15.538073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:15.538895 (+ 822us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:15.539538 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:15.539553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:15.776715 (+237162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:15.779713 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:15.779731 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:15.779755 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:15.780239 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:15.780622 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:15.780628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:15.780633 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:15.780895 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:15.781438 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:15.781450 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:15.781894 (+ 444us) client_negotiation.cc:769] Sending connection context
0929 10:23:15.782148 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 10:23:15.782573 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":154,"threads_started":1}
I20260929 10:23:15.785369 22404 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36119
I20260929 10:23:15.785681 22404 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:15.786404 22404 heartbeater.cc:511] Master 127.21.33.190:36119 requested a full tablet report, sending...
I20260929 10:23:15.788058 22212 ts_manager.cc:194] Registered new tserver with Master: 0346ef7182394e2ba794f32ec9d2bc63 (127.21.33.129:42509)
I20260929 10:23:15.789324 22212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:57955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
W20260929 10:23:15.928714 21638 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.33.130: real 0.396s user 0.000s sys 0.004s
I20260929 10:23:15.929276 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36119
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:16.039566 22410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:16.039885 22410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:16.039963 22410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:16.044000 22410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:16.044102 22410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:16.044201 22410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:16.049548 22410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:16.050688 22410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:16.051752 22410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:16.061385 22416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:16.061548 22410 server_base.cc:1034] running on GCE node
W20260929 10:23:16.061316 22415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:16.061342 22418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:16.062211 22410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:16.062777 22410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:16.063936 22410 hybrid_clock.cc:648] HybridClock initialized: now 1790677396063924 us; error 25 us; skew 500 ppm
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:16.066711 22410 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:16.067574 22410 webserver.cc:460] Webserver started at http://127.21.33.130:42973/ using document root <none> and password file <none>
I20260929 10:23:16.068166 22410 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:16.068257 22410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:16.068486 22410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:16.070571 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4"
format_stamp: "Formatted at 2026-09-29 10:23:16 on dist-test-slave-2rbl"
server_key: "b1f459443139e94ebfa6f8cbb5835e61"
server_key_iv: "5caff5a6f4bdb0cc484114f57669060b"
server_key_version: "encryptionkey@0"
I20260929 10:23:16.071156 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4"
format_stamp: "Formatted at 2026-09-29 10:23:16 on dist-test-slave-2rbl"
server_key: "b1f459443139e94ebfa6f8cbb5835e61"
server_key_iv: "5caff5a6f4bdb0cc484114f57669060b"
server_key_version: "encryptionkey@0"
I20260929 10:23:16.074895 22410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 10:23:16.077337 22425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:16.078529 22410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:23:16.078670 22410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4"
format_stamp: "Formatted at 2026-09-29 10:23:16 on dist-test-slave-2rbl"
server_key: "b1f459443139e94ebfa6f8cbb5835e61"
server_key_iv: "5caff5a6f4bdb0cc484114f57669060b"
server_key_version: "encryptionkey@0"
I20260929 10:23:16.078784 22410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:16.101775 22410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:16.104700 22410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:16.104905 22410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:16.105559 22410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:16.106379 22410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:16.106449 22410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:16.106516 22410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:16.106565 22410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:16.115742 22410 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:44143
I20260929 10:23:16.115767 22539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:44143 every 8 connection(s)
I20260929 10:23:16.116688 22410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:16.125735 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22410
I20260929 10:23:16.125833 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:16.126116 21638 external_mini_cluster.cc:1428] Setting key 9bde736e1b13c364958cd2e19fa9744b
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:16.130868 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.118489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:32937 (local address 127.21.33.190:36119)
0929 10:23:16.118623 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.118627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.119471 (+ 844us) server_negotiation.cc:407] Connection header received
0929 10:23:16.120865 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.120869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.120933 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.121062 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:16.122354 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.122905 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.124101 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.124353 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.124481 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.127261 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:16.127264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:16.127266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:16.127290 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:16.128738 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.129312 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.129316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.129318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.129399 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.129764 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.129767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.129769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.129946 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:16.130050 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.130548 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.130676 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:16.131650 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.118804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36119 (local address 127.21.33.130:32937)
0929 10:23:16.119295 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:16.119340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:16.120632 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:16.121263 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:16.121274 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:16.121701 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:16.122192 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.122206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.123044 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.123048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.123959 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.123968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.124520 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.124525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.124711 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.125421 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:16.125438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:16.127080 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:16.128874 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:16.128880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:16.128895 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:16.129150 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:16.129502 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:16.129505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:16.129507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:16.129655 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:16.130060 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:16.130068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:16.130314 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 10:23:16.130498 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 10:23:16.130821 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260929 10:23:16.132787 22540 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36119
I20260929 10:23:16.133038 22540 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:16.133538 22540 heartbeater.cc:511] Master 127.21.33.190:36119 requested a full tablet report, sending...
I20260929 10:23:16.134593 22212 ts_manager.cc:194] Registered new tserver with Master: b06a45f45cf74ff3bcedbc0cf8e059a4 (127.21.33.130:44143)
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:16.135407 22212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:32937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:16.152470 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36119
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:16.265491 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:16.265748 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:16.265791 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:16.269912 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:16.269969 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:16.270028 22545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:16.275249 22545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:16.276265 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:16.277314 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:16.287128 22551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:16.287220 22553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:16.287144 22550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:16.287516 22545 server_base.cc:1034] running on GCE node
I20260929 10:23:16.287905 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:16.288565 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:16.289733 22545 hybrid_clock.cc:648] HybridClock initialized: now 1790677396289700 us; error 44 us; skew 500 ppm
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:16.292629 22545 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:16.293522 22545 webserver.cc:460] Webserver started at http://127.21.33.131:38779/ using document root <none> and password file <none>
I20260929 10:23:16.294088 22545 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:16.294186 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:16.294412 22545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:16.296407 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea"
format_stamp: "Formatted at 2026-09-29 10:23:16 on dist-test-slave-2rbl"
server_key: "d6d3af24e2c3886bc10c5012c2656249"
server_key_iv: "3f673abb322a9f4423af39a996998b73"
server_key_version: "encryptionkey@0"
I20260929 10:23:16.296972 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea"
format_stamp: "Formatted at 2026-09-29 10:23:16 on dist-test-slave-2rbl"
server_key: "d6d3af24e2c3886bc10c5012c2656249"
server_key_iv: "3f673abb322a9f4423af39a996998b73"
server_key_version: "encryptionkey@0"
I20260929 10:23:16.300601 22545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:16.303081 22560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:16.304311 22545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:16.304443 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea"
format_stamp: "Formatted at 2026-09-29 10:23:16 on dist-test-slave-2rbl"
server_key: "d6d3af24e2c3886bc10c5012c2656249"
server_key_iv: "3f673abb322a9f4423af39a996998b73"
server_key_version: "encryptionkey@0"
I20260929 10:23:16.304550 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:16.344743 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:16.347930 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:16.348168 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:16.348783 22545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:16.349720 22545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:16.349800 22545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:16.349874 22545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:16.349908 22545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:16.358682 22545 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:33803
I20260929 10:23:16.358716 22673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:33803 every 8 connection(s)
I20260929 10:23:16.359659 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:16.369467 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22545
I20260929 10:23:16.369614 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:16.369932 21638 external_mini_cluster.cc:1428] Setting key fcf9850ec8e9a241eb267a38e84f4863
I20260929 10:23:16.374330 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.361597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:48375 (local address 127.21.33.190:36119)
0929 10:23:16.361779 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.361783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.362748 (+ 965us) server_negotiation.cc:407] Connection header received
0929 10:23:16.364012 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.364015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.364065 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.364156 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:16.365453 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.366032 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.367190 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.367514 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.367675 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.370464 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:16.370468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:16.370470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:16.370507 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:16.372207 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.372731 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.372736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.372738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.372821 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.373191 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.373220 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.373222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.373412 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:16.373521 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.374042 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.374163 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 10:23:16.375151 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.362082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36119 (local address 127.21.33.131:48375)
0929 10:23:16.362560 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:16.362603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:16.363804 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:16.364324 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:16.364331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:16.364729 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:16.365244 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.365258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.366164 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.366166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.367053 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.367062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.367726 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.367729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.367876 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.368534 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:16.368550 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:16.370277 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:16.372322 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:16.372329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:16.372346 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:16.372618 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:16.372915 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:16.372920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:16.372922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:16.373077 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:16.373519 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:16.373526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:16.373796 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 10:23:16.373988 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 10:23:16.374330 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260929 10:23:16.376240 22674 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36119
I20260929 10:23:16.376519 22674 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:16.377056 22674 heartbeater.cc:511] Master 127.21.33.190:36119 requested a full tablet report, sending...
I20260929 10:23:16.378139 22212 ts_manager.cc:194] Registered new tserver with Master: 8f91c7531a9e4eeda3c1c03c2d58ceea (127.21.33.131:33803)
I20260929 10:23:16.378732 22212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:48375
I20260929 10:23:16.384049 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:23:16.395573 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.387646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.21.33.190:36119)
0929 10:23:16.387919 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.387924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.388070 (+ 146us) server_negotiation.cc:407] Connection header received
0929 10:23:16.388238 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.388248 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.388289 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.388378 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:16.389088 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.389756 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.390620 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.390926 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.391037 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.392507 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:16.392511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:16.392516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:16.392543 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:16.393875 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.394305 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.394310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.394312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.394395 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.394674 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.394678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.394679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.394903 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:16.395048 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.395292 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.395408 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260929 10:23:16.400801 22212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:16.403986 22212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:16.419234 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.413994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.21.33.129:42509)
0929 10:23:16.414393 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.414400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.414424 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:23:16.414500 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.414506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.414741 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.414895 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:16.415236 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.416400 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.417915 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.418579 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.418714 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.418865 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.419000 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.419094 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":72,"threads_started":1}
I20260929 10:23:16.419234 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.413217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42509 (local address 127.0.0.1:39216)
0929 10:23:16.413991 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:16.414184 (+ 193us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:16.414274 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:16.414882 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:16.414889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:16.414915 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:16.415098 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.415106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.416534 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.416537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.417779 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.417789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.418705 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.418708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.418809 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.418844 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 10:23:16.418959 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 10:23:16.419078 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260929 10:23:16.419584 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.413307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.21.33.130:44143)
0929 10:23:16.414562 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.414577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.414604 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:23:16.414686 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.414693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.414882 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.415314 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:16.415482 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.416302 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.417864 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.418607 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.418723 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.418868 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.419324 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.419419 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":124,"threads_started":1}
I20260929 10:23:16.419610 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.413146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:44143 (local address 127.0.0.1:50398)
0929 10:23:16.413861 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:16.413916 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:16.414051 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:16.415035 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:16.415038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:16.415068 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:16.415170 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.415179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.416443 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.416450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.417727 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.417737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.418940 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.418944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.419065 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.419093 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:23:16.419169 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:23:16.419243 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":55,"thread_start_us":151,"threads_started":1}
I20260929 10:23:16.421497 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.413997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.21.33.131:33803)
0929 10:23:16.415046 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.415066 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.415083 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:23:16.415141 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.415145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.415342 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.415488 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:16.415773 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.416826 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.418266 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.419082 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.419193 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.419321 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.421314 (+ 1993us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.421391 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":117,"threads_started":1}
I20260929 10:23:16.421689 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.413455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:33803 (local address 127.0.0.1:59584)
0929 10:23:16.414328 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:16.414342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:16.414423 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:16.415489 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:16.415492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:16.415513 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:16.415640 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.415648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.416998 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.417001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.418133 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.418142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.419383 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.419387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.419472 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.419497 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:23:16.419597 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 10:23:16.419681 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260929 10:23:16.422601 22473 tablet_service.cc:1467] Processing CreateTablet for tablet fd63529dcf824614aff518287276ab8f (DEFAULT_TABLE table=test-table [id=16f15869a52d4b15b2afbe8ba2c91717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:16.422602 22338 tablet_service.cc:1467] Processing CreateTablet for tablet fd63529dcf824614aff518287276ab8f (DEFAULT_TABLE table=test-table [id=16f15869a52d4b15b2afbe8ba2c91717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:16.423777 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd63529dcf824614aff518287276ab8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:16.423777 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd63529dcf824614aff518287276ab8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:16.424153 22608 tablet_service.cc:1467] Processing CreateTablet for tablet fd63529dcf824614aff518287276ab8f (DEFAULT_TABLE table=test-table [id=16f15869a52d4b15b2afbe8ba2c91717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:16.425037 22608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd63529dcf824614aff518287276ab8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:16.430657 22693 tablet_bootstrap.cc:492] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea: Bootstrap starting.
I20260929 10:23:16.431973 22695 tablet_bootstrap.cc:492] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4: Bootstrap starting.
I20260929 10:23:16.432862 22693 tablet_bootstrap.cc:654] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:16.433545 22694 tablet_bootstrap.cc:492] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63: Bootstrap starting.
I20260929 10:23:16.434090 22693 log.cc:824] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:16.434396 22695 tablet_bootstrap.cc:654] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:16.435231 22695 log.cc:824] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:16.435827 22694 tablet_bootstrap.cc:654] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:16.436117 22693 tablet_bootstrap.cc:492] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea: No bootstrap required, opened a new log
I20260929 10:23:16.436318 22693 ts_tablet_manager.cc:1397] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 10:23:16.436663 22694 log.cc:824] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:16.437306 22695 tablet_bootstrap.cc:492] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4: No bootstrap required, opened a new log
I20260929 10:23:16.437511 22695 ts_tablet_manager.cc:1397] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:23:16.438580 22694 tablet_bootstrap.cc:492] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63: No bootstrap required, opened a new log
I20260929 10:23:16.438757 22694 ts_tablet_manager.cc:1397] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:16.440284 22693 raft_consensus.cc:348] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.440608 22693 raft_consensus.cc:374] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:16.440728 22693 raft_consensus.cc:729] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f91c7531a9e4eeda3c1c03c2d58ceea, State: Initialized, Role: FOLLOWER
I20260929 10:23:16.440936 22695 raft_consensus.cc:348] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.441253 22695 raft_consensus.cc:374] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:16.441218 22693 consensus_queue.cc:260] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.441322 22695 raft_consensus.cc:729] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06a45f45cf74ff3bcedbc0cf8e059a4, State: Initialized, Role: FOLLOWER
I20260929 10:23:16.441785 22695 consensus_queue.cc:260] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.442243 22693 ts_tablet_manager.cc:1428] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:23:16.442520 22540 heartbeater.cc:503] Master 127.21.33.190:36119 was elected leader, sending a full tablet report...
I20260929 10:23:16.442646 22674 heartbeater.cc:503] Master 127.21.33.190:36119 was elected leader, sending a full tablet report...
I20260929 10:23:16.442898 22694 raft_consensus.cc:348] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.443080 22694 raft_consensus.cc:374] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:16.443110 22694 raft_consensus.cc:729] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0346ef7182394e2ba794f32ec9d2bc63, State: Initialized, Role: FOLLOWER
I20260929 10:23:16.443447 22694 consensus_queue.cc:260] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.444032 22695 ts_tablet_manager.cc:1428] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:23:16.444067 22404 heartbeater.cc:503] Master 127.21.33.190:36119 was elected leader, sending a full tablet report...
I20260929 10:23:16.444377 22694 ts_tablet_manager.cc:1428] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:23:16.444768 22699 raft_consensus.cc:482] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:16.444914 22699 raft_consensus.cc:504] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.446198 22699 leader_election.cc:288] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b06a45f45cf74ff3bcedbc0cf8e059a4 (127.21.33.130:44143), 0346ef7182394e2ba794f32ec9d2bc63 (127.21.33.129:42509)
I20260929 10:23:16.451256 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.446968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:38705 (local address 127.21.33.130:44143)
0929 10:23:16.447160 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.447164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.447176 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:16.447228 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.447231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.447272 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.447347 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:16.447733 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.448431 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.449952 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.450595 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.450729 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.450786 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.451058 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.451124 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260929 10:23:16.451256 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.446915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:41975 (local address 127.21.33.129:42509)
0929 10:23:16.447025 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.447030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.447140 (+ 110us) server_negotiation.cc:407] Connection header received
0929 10:23:16.447338 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.447342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.447391 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.447442 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:16.447901 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.448579 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.449959 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.450602 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.450732 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.450786 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.451052 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.451124 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:23:16.451323 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.446359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:44143 (local address 127.21.33.131:38705)
0929 10:23:16.446583 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:16.446599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:16.446676 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:16.447466 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:16.447471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:16.447488 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:16.447597 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.447607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.448534 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.448539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.449802 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.449812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.450734 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.450737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.450902 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.450941 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:23:16.451019 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:23:16.451199 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":103}
I20260929 10:23:16.451340 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.446516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42509 (local address 127.21.33.131:41975)
0929 10:23:16.447047 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:16.447064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:16.447208 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:16.447608 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:16.447611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:16.447631 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:16.447761 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.447768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.448712 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.448715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.449855 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:16.449865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.450724 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.450727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:16.450899 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.450947 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 10:23:16.451019 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:23:16.451160 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":90,"thread_start_us":111,"threads_started":1}
I20260929 10:23:16.451910 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd63529dcf824614aff518287276ab8f" candidate_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0346ef7182394e2ba794f32ec9d2bc63" is_pre_election: true
I20260929 10:23:16.452167 22358 raft_consensus.cc:2454] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f91c7531a9e4eeda3c1c03c2d58ceea in term 0.
I20260929 10:23:16.452209 22493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd63529dcf824614aff518287276ab8f" candidate_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" is_pre_election: true
I20260929 10:23:16.452502 22493 raft_consensus.cc:2454] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f91c7531a9e4eeda3c1c03c2d58ceea in term 0.
I20260929 10:23:16.452651 22563 leader_election.cc:302] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0346ef7182394e2ba794f32ec9d2bc63, 8f91c7531a9e4eeda3c1c03c2d58ceea; no voters:
I20260929 10:23:16.452940 22699 raft_consensus.cc:2781] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:16.453043 22699 raft_consensus.cc:482] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:16.453109 22699 raft_consensus.cc:3037] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:16.454166 22699 raft_consensus.cc:504] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.454528 22699 leader_election.cc:288] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [CANDIDATE]: Term 1 election: Requested vote from peers b06a45f45cf74ff3bcedbc0cf8e059a4 (127.21.33.130:44143), 0346ef7182394e2ba794f32ec9d2bc63 (127.21.33.129:42509)
I20260929 10:23:16.454864 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd63529dcf824614aff518287276ab8f" candidate_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0346ef7182394e2ba794f32ec9d2bc63"
I20260929 10:23:16.455044 22358 raft_consensus.cc:3037] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:16.456180 22358 raft_consensus.cc:2454] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f91c7531a9e4eeda3c1c03c2d58ceea in term 1.
I20260929 10:23:16.456477 22493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd63529dcf824614aff518287276ab8f" candidate_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4"
I20260929 10:23:16.456517 22563 leader_election.cc:302] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0346ef7182394e2ba794f32ec9d2bc63, 8f91c7531a9e4eeda3c1c03c2d58ceea; no voters:
I20260929 10:23:16.456614 22493 raft_consensus.cc:3037] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:16.456677 22699 raft_consensus.cc:2781] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:16.456899 22699 raft_consensus.cc:686] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [term 1 LEADER]: Becoming Leader. State: Replica: 8f91c7531a9e4eeda3c1c03c2d58ceea, State: Running, Role: LEADER
I20260929 10:23:16.457266 22699 consensus_queue.cc:237] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } }
I20260929 10:23:16.457912 22493 raft_consensus.cc:2454] T fd63529dcf824614aff518287276ab8f P b06a45f45cf74ff3bcedbc0cf8e059a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f91c7531a9e4eeda3c1c03c2d58ceea in term 1.
I20260929 10:23:16.460482 22211 catalog_manager.cc:5495] T fd63529dcf824614aff518287276ab8f P 8f91c7531a9e4eeda3c1c03c2d58ceea reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f91c7531a9e4eeda3c1c03c2d58ceea (127.21.33.131). New cstate: current_term: 1 leader_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06a45f45cf74ff3bcedbc0cf8e059a4" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0346ef7182394e2ba794f32ec9d2bc63" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f91c7531a9e4eeda3c1c03c2d58ceea" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33803 } health_report { overall_health: HEALTHY } } }
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:16.479539 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.470869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.21.33.129:42509)
0929 10:23:16.471043 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.471048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.471207 (+ 159us) server_negotiation.cc:407] Connection header received
0929 10:23:16.471376 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.471380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.471431 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.471521 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:16.472184 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.472685 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.473640 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.473868 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.473980 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.476361 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:16.476372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:16.476384 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:16.476406 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:16.478045 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.478462 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.478468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.478469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.478547 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.478793 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.478795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.478796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.479030 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:16.479204 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.479331 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.479421 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:16.506479 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.497776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.21.33.129:42509)
0929 10:23:16.497981 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.497985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.498015 (+ 30us) server_negotiation.cc:407] Connection header received
0929 10:23:16.498149 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.498152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.498193 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.498263 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:16.498943 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.499483 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.500327 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.500633 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.500730 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.503149 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:16.503151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:16.503153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:16.503175 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:16.504635 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.505062 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.505066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.505067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.505165 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.505519 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.505522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.505523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.505704 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:16.505816 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.506140 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.506306 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
W20260929 10:23:16.530180 22405 tablet.cc:2369] T fd63529dcf824614aff518287276ab8f P 0346ef7182394e2ba794f32ec9d2bc63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:16.534046 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:16.524187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.21.33.129:42509)
0929 10:23:16.524398 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:16.524403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:16.524482 (+ 79us) server_negotiation.cc:407] Connection header received
0929 10:23:16.524669 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:16.524674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:16.524726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:16.524805 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:16.525867 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.526677 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.527680 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:16.527992 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:16.528101 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:16.530858 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:16.530861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:16.530865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:16.530887 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:16.532446 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.532869 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.532872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.532875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.532962 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:16.533232 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:16.533235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:16.533237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:16.533422 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:16.533550 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:16.533779 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 10:23:16.533896 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260929 10:23:16.535446 22314 tablet_service.cc:2908] Rejecting scan request for tablet fd63529dcf824614aff518287276ab8f: Uninitialized: clean time has not yet been initialized
I20260929 10:23:16.537479 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22274
I20260929 10:23:16.543869 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22410
I20260929 10:23:16.549314 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22545
I20260929 10:23:16.555256 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22180
2026-09-29T10:23:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (6526 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22717](info): set up 2 sockets
Sep 29 10:23:16 dist-test-slave-2rbl krb5kdc[22717](info): commencing operation
krb5kdc: starting...
W20260929 10:23:17.597842 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:17 dist-test-slave-2rbl krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:17.724548 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33599
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:33599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:17.835671 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:17.835937 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:17.835976 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:17.840065 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:17.840124 22730 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:23:17.840142 22730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:17.840159 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:17.840175 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:17.840191 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:17.845507 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:33599
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:17.846573 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:17.847628 22730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:17.856289 22738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:17.856233 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:17.856333 22730 server_base.cc:1034] running on GCE node
W20260929 10:23:17.856240 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:17.857139 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:17.858311 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:17.859505 22730 hybrid_clock.cc:648] HybridClock initialized: now 1790677397859491 us; error 28 us; skew 500 ppm
Sep 29 10:23:17 dist-test-slave-2rbl krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677397, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:17.862354 22730 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:17.863294 22730 webserver.cc:460] Webserver started at http://127.21.33.190:46539/ using document root <none> and password file <none>
I20260929 10:23:17.863860 22730 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:17.863952 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:17.864193 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:17.866333 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "4c471b20f38a4933936d80ac5869ad76"
format_stamp: "Formatted at 2026-09-29 10:23:17 on dist-test-slave-2rbl"
server_key: "e09e64457096e6f8b43fd6433f63aa92"
server_key_iv: "4e615cc18e1ade7cd81a98f5d0b064da"
server_key_version: "encryptionkey@0"
I20260929 10:23:17.866899 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "4c471b20f38a4933936d80ac5869ad76"
format_stamp: "Formatted at 2026-09-29 10:23:17 on dist-test-slave-2rbl"
server_key: "e09e64457096e6f8b43fd6433f63aa92"
server_key_iv: "4e615cc18e1ade7cd81a98f5d0b064da"
server_key_version: "encryptionkey@0"
I20260929 10:23:17.870886 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:23:17.873462 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:17.874725 22730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:17.874882 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "4c471b20f38a4933936d80ac5869ad76"
format_stamp: "Formatted at 2026-09-29 10:23:17 on dist-test-slave-2rbl"
server_key: "e09e64457096e6f8b43fd6433f63aa92"
server_key_iv: "4e615cc18e1ade7cd81a98f5d0b064da"
server_key_version: "encryptionkey@0"
I20260929 10:23:17.875002 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:17.896301 22730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:17.899773 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:17.900031 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:17.908857 22730 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:33599
I20260929 10:23:17.908869 22807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:33599 every 8 connection(s)
I20260929 10:23:17.909981 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:17.911152 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22730
I20260929 10:23:17.911273 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:17.911535 21638 external_mini_cluster.cc:1428] Setting key cab44e6f5abcccd29e15fc69154980b8
I20260929 10:23:17.913314 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:17.920085 22808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76: Bootstrap starting.
Sep 29 10:23:17 dist-test-slave-2rbl krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:17.922693 22808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:17.924023 22808 log.cc:824] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:17.926393 22808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76: No bootstrap required, opened a new log
I20260929 10:23:17.926906 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:17.912998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.21.33.190:33599)
0929 10:23:17.913442 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:17.913461 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:17.913503 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:23:17.914332 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:17.914354 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:17.914757 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:17.915127 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:17.915773 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:17.917300 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:17.918130 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:17.918828 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:17.918997 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:17.921608 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:17.921615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:17.921631 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:17.921662 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:17.923681 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:17.924126 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:17.924132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:17.924138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:17.924274 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:17.924623 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:17.924627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:17.924629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:17.925132 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:17.925337 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:17.925655 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 10:23:17.925952 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":159,"threads_started":1}
I20260929 10:23:17.929927 22808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c471b20f38a4933936d80ac5869ad76" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33599 } }
I20260929 10:23:17.930140 22808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:17.930174 22808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c471b20f38a4933936d80ac5869ad76, State: Initialized, Role: FOLLOWER
I20260929 10:23:17.930625 22808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c471b20f38a4933936d80ac5869ad76" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33599 } }
I20260929 10:23:17.930801 22808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:17.930894 22808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:17.931003 22808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:17.931900 22808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c471b20f38a4933936d80ac5869ad76" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33599 } }
I20260929 10:23:17.932191 22808 leader_election.cc:302] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c471b20f38a4933936d80ac5869ad76; no voters:
I20260929 10:23:17.932438 22808 leader_election.cc:288] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:17.932574 22813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:17.932794 22813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [term 1 LEADER]: Becoming Leader. State: Replica: 4c471b20f38a4933936d80ac5869ad76, State: Running, Role: LEADER
I20260929 10:23:17.933132 22813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c471b20f38a4933936d80ac5869ad76" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33599 } }
I20260929 10:23:17.933725 22808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:17.934705 22814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c471b20f38a4933936d80ac5869ad76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c471b20f38a4933936d80ac5869ad76" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33599 } } }
I20260929 10:23:17.934840 22814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:17.935231 22815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c471b20f38a4933936d80ac5869ad76. Latest consensus state: current_term: 1 leader_uuid: "4c471b20f38a4933936d80ac5869ad76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c471b20f38a4933936d80ac5869ad76" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33599 } } }
I20260929 10:23:17.935389 22815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:17.936158 22822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:17.938906 22822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:17.944271 22822 catalog_manager.cc:1347] Generated new cluster ID: 7101549e48f44c3f9636795cd309e135
I20260929 10:23:17.944356 22822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:17.963222 22822 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:17.964301 22822 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:17.969695 22822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76: Generated new TSK 0
I20260929 10:23:17.970474 22822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:18.008319 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33599
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:18.121222 22834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:18.121486 22834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:18.121532 22834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:18.125547 22834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:18.125604 22834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:18.125643 22834 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:18.125716 22834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:18.130942 22834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:33599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:18.131999 22834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:18.133020 22834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:18.142767 22840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:18.142777 22839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:18.142799 22842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:18.143180 22834 server_base.cc:1034] running on GCE node
I20260929 10:23:18.143535 22834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:18.144200 22834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:18.145363 22834 hybrid_clock.cc:648] HybridClock initialized: now 1790677398145325 us; error 50 us; skew 500 ppm
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:18.148173 22834 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:18.149117 22834 webserver.cc:460] Webserver started at http://127.21.33.129:37005/ using document root <none> and password file <none>
I20260929 10:23:18.149772 22834 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:18.149943 22834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:18.150211 22834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:18.152515 22834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "2cbb2e82947a4d60b18b4cb07e002709"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "923bf444ce0eea88fe536f014b5993f9"
server_key_iv: "8b4ae29bf840345d89d8960501fe2534"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.153422 22834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "2cbb2e82947a4d60b18b4cb07e002709"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "923bf444ce0eea88fe536f014b5993f9"
server_key_iv: "8b4ae29bf840345d89d8960501fe2534"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.157459 22834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:23:18.160149 22849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:18.161412 22834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:18.161544 22834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "2cbb2e82947a4d60b18b4cb07e002709"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "923bf444ce0eea88fe536f014b5993f9"
server_key_iv: "8b4ae29bf840345d89d8960501fe2534"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.161692 22834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:18.191582 22834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:18.194911 22834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:18.195127 22834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:18.196564 22834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:18.196646 22834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:18.196720 22834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:18.196772 22834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:18.207898 22834 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:37331
I20260929 10:23:18.208256 22964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:37331 every 8 connection(s)
I20260929 10:23:18.209031 22834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:18.210953 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.197564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:35809 (local address 127.21.33.190:33599)
0929 10:23:18.197741 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.197745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.198946 (+ 1201us) server_negotiation.cc:407] Connection header received
0929 10:23:18.200240 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.200246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.200315 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.200424 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:18.201839 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.202389 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.203544 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.203823 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.203947 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.206916 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:18.206919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:18.206921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:18.206945 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:18.208759 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.209294 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.209298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.209300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.209404 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.209800 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.209804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.209806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.210043 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:18.210174 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.210649 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.210782 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 10:23:18.211710 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.198159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33599 (local address 127.21.33.129:35809)
0929 10:23:18.198699 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.198742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.199979 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.200630 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.200639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.201135 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:18.201682 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.201695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.202530 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.202534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.203413 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.203422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.203994 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.203998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.204171 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.204830 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:18.204849 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:18.206732 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:18.208866 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.208871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.208882 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.209165 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.209506 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.209510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.209513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.209681 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.210170 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:18.210175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:18.210412 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.210619 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.210933 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":154,"threads_started":1}
I20260929 10:23:18.212958 22965 heartbeater.cc:347] Connected to a master server at 127.21.33.190:33599
I20260929 10:23:18.213127 22965 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:18.213655 22965 heartbeater.cc:511] Master 127.21.33.190:33599 requested a full tablet report, sending...
I20260929 10:23:18.215188 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22834
I20260929 10:23:18.215313 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:18.215343 22761 ts_manager.cc:194] Registered new tserver with Master: 2cbb2e82947a4d60b18b4cb07e002709 (127.21.33.129:37331)
I20260929 10:23:18.215564 21638 external_mini_cluster.cc:1428] Setting key b811de6ee424c0a2d479452b6173b9d3
I20260929 10:23:18.216640 22761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:35809
I20260929 10:23:18.225087 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.215213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:35489 (local address 127.21.33.190:33599)
0929 10:23:18.217237 (+ 2024us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.217242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.217262 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:23:18.217290 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.217292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.217324 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.217410 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:18.217885 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.218402 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.219787 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.220011 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.220109 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.221582 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:18.221587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:18.221590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:18.221616 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:18.223245 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.223753 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.223758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.223760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.223861 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.224172 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.224178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.224180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.224374 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:18.224490 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.224781 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.224903 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1932}
I20260929 10:23:18.225098 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.215174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33599 (local address 127.21.33.129:35489)
0929 10:23:18.215450 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.215465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.215551 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.217431 (+ 1880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.217441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.217655 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:18.217770 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.217776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.218540 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.218543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.219637 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.219649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.220147 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.220153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.220280 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.220872 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:18.220893 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:18.221414 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:18.223391 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.223395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.223398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.223637 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.223963 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.223968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.223970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.224071 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.224480 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:18.224483 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:18.224604 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.224723 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.224893 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:18.245326 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33599
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:18.356487 22974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:18.356763 22974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:18.356808 22974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:18.360841 22974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:18.360906 22974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:18.360944 22974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:18.361016 22974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:18.366501 22974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:33599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:18.367637 22974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:18.368710 22974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:18.378662 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:18.378716 22974 server_base.cc:1034] running on GCE node
W20260929 10:23:18.378662 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:18.378690 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:18.379400 22974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:18.380144 22974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:18.381305 22974 hybrid_clock.cc:648] HybridClock initialized: now 1790677398381264 us; error 44 us; skew 500 ppm
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:18.384272 22974 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:18.385324 22974 webserver.cc:460] Webserver started at http://127.21.33.130:35171/ using document root <none> and password file <none>
I20260929 10:23:18.385933 22974 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:18.386029 22974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:18.386278 22974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:18.388454 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "08a6b3760be4423fb32807487abab116"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "7452fe96179e987b1420bd7c17c5876e"
server_key_iv: "b21e309adfaf18bbb39f068d95fd7454"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.389057 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "08a6b3760be4423fb32807487abab116"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "7452fe96179e987b1420bd7c17c5876e"
server_key_iv: "b21e309adfaf18bbb39f068d95fd7454"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.393138 22974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 10:23:18.395953 22990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:18.397364 22974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:23:18.397508 22974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "08a6b3760be4423fb32807487abab116"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "7452fe96179e987b1420bd7c17c5876e"
server_key_iv: "b21e309adfaf18bbb39f068d95fd7454"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.397675 22974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:18.422638 22974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:18.425954 22974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:18.426167 22974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:18.427610 22974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:18.427686 22974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:18.427759 22974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:18.427809 22974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:18.439430 22974 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:34117
I20260929 10:23:18.439623 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:34117 every 8 connection(s)
I20260929 10:23:18.440382 22974 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:18.442332 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 22974
I20260929 10:23:18.442420 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:18.442680 21638 external_mini_cluster.cc:1428] Setting key 5e78d4bc3db4b2513e0a97563defad44
I20260929 10:23:18.442914 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.428541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:59813 (local address 127.21.33.190:33599)
0929 10:23:18.428699 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.428703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.429793 (+ 1090us) server_negotiation.cc:407] Connection header received
0929 10:23:18.430941 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.430948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.431021 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.431128 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:18.432690 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.433507 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.434927 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.435292 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.435423 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.438834 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:18.438837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:18.438839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:18.438865 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:18.440655 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.441307 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.441311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.441312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.441407 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.441776 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.441779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.441781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.441979 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:18.442094 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.442612 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.442742 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:23:18.443811 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.429130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33599 (local address 127.21.33.130:59813)
0929 10:23:18.429641 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.429684 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.430692 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.431341 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.431351 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.431808 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:18.432503 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.432523 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.433643 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.433646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.434798 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.434808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.435472 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.435476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.435710 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.436440 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:18.436461 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:18.438632 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:18.440801 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.440806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.440822 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.441184 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.441503 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.441507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.441510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.441662 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.442090 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:18.442096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:18.442347 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.442553 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.442876 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260929 10:23:18.444957 23106 heartbeater.cc:347] Connected to a master server at 127.21.33.190:33599
I20260929 10:23:18.445122 23106 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:18.445633 23106 heartbeater.cc:511] Master 127.21.33.190:33599 requested a full tablet report, sending...
I20260929 10:23:18.446651 22761 ts_manager.cc:194] Registered new tserver with Master: 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117)
I20260929 10:23:18.447294 22761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:59813
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:18.455368 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.447847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:52385 (local address 127.21.33.190:33599)
0929 10:23:18.447955 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.447959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.447971 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:18.448007 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.448011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.448051 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.448130 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:18.448732 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.449622 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.450454 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.450852 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.450969 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.452030 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:18.452036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:18.452039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:18.452069 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:18.453747 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.454148 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.454151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.454152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.454232 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.454546 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.454549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.454551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.454805 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:18.454899 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.455125 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.455215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:23:18.455358 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.447444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33599 (local address 127.21.33.130:52385)
0929 10:23:18.447755 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.447773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.447864 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.448175 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.448180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.448415 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:18.448573 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.448584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.449771 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.449774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.450335 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.450344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.450973 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.450976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.451043 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.451580 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:18.451591 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:18.451891 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:18.453851 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.453853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.453854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.454042 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.454337 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.454339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.454341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.454424 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.454920 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:18.454922 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:18.455000 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.455089 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.455227 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:18.473160 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33599
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:18.583714 23115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:18.584002 23115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:18.584077 23115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:18.588096 23115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:18.588177 23115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:18.588250 23115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:18.588330 23115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:18.593490 23115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:33599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:18.594564 23115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:18.595630 23115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:18.605510 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:18.605608 23115 server_base.cc:1034] running on GCE node
W20260929 10:23:18.605554 23123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:18.605557 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:18.606110 23115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:18.606814 23115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:18.608104 23115 hybrid_clock.cc:648] HybridClock initialized: now 1790677398608099 us; error 52 us; skew 500 ppm
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:18.610924 23115 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:18.611714 23115 webserver.cc:460] Webserver started at http://127.21.33.131:42647/ using document root <none> and password file <none>
I20260929 10:23:18.612223 23115 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:18.612279 23115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:18.612442 23115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:18.614605 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "0e618cdc16084bb985c3aa0798056d60"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "43538e05aa0e8e7d75c1e2d8fb163930"
server_key_iv: "2a43ab966d60b2ad59585c0236534c01"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.615123 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "0e618cdc16084bb985c3aa0798056d60"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "43538e05aa0e8e7d75c1e2d8fb163930"
server_key_iv: "2a43ab966d60b2ad59585c0236534c01"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.618737 23115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:18.621351 23130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:18.622644 23115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:18.622745 23115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "0e618cdc16084bb985c3aa0798056d60"
format_stamp: "Formatted at 2026-09-29 10:23:18 on dist-test-slave-2rbl"
server_key: "43538e05aa0e8e7d75c1e2d8fb163930"
server_key_iv: "2a43ab966d60b2ad59585c0236534c01"
server_key_version: "encryptionkey@0"
I20260929 10:23:18.622872 23115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:18.634522 23115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:18.637940 23115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:18.638146 23115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:18.639608 23115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:18.639676 23115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:18.639740 23115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:18.639776 23115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:18.650767 23115 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:45213
I20260929 10:23:18.650785 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:45213 every 8 connection(s)
I20260929 10:23:18.651700 23115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:18.654309 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.640467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:53703 (local address 127.21.33.190:33599)
0929 10:23:18.640600 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.640603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.641668 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 10:23:18.642856 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.642860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.642924 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.643037 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:18.644606 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.645591 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.647008 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.647335 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.647457 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.650501 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:18.650505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:18.650510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:18.650543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:18.652148 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.652728 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.652731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.652733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.652825 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.653166 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.653170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.653171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.653405 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:18.653514 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.654008 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.654139 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:23:18.655042 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.640982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33599 (local address 127.21.33.131:53703)
0929 10:23:18.641486 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.641528 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.642567 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.643270 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.643281 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.643736 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:18.644422 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.644441 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.645750 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.645775 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.646880 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.646889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.647496 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.647501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.647687 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.648318 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:18.648334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:18.650302 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:18.652288 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.652293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.652308 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.652606 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.652917 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.652922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.652924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.653062 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.653521 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:18.653530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:18.653768 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.653951 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.654260 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":151,"threads_started":1}
I20260929 10:23:18.656157 23247 heartbeater.cc:347] Connected to a master server at 127.21.33.190:33599
I20260929 10:23:18.656312 23247 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:18.656805 23247 heartbeater.cc:511] Master 127.21.33.190:33599 requested a full tablet report, sending...
I20260929 10:23:18.657886 22762 ts_manager.cc:194] Registered new tserver with Master: 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131:45213)
I20260929 10:23:18.658699 22762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:53703
I20260929 10:23:18.659256 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 23115
I20260929 10:23:18.659390 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:18.659682 21638 external_mini_cluster.cc:1428] Setting key 6979a42f8024a4575febc8f2d13c131a
I20260929 10:23:18.661734 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:23:18.666383 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.658458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:44577 (local address 127.21.33.190:33599)
0929 10:23:18.658565 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.658598 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.658802 (+ 204us) server_negotiation.cc:407] Connection header received
0929 10:23:18.658900 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.658902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.658963 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.659037 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:18.659631 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.660368 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.661387 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.661700 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.661811 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.662939 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:18.662942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:18.662945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:18.662973 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:18.664685 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.665090 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.665097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.665101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.665224 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.665500 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.665504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.665505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.665766 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:18.665876 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.666116 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.666221 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:23:18.666397 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.658408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33599 (local address 127.21.33.131:44577)
0929 10:23:18.658698 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.658714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.658812 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.659147 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.659149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.659349 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:18.659465 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.659472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.660524 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.660527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.661268 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.661278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.661812 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.661815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.661908 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.662428 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:18.662440 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:18.662782 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:18.664789 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.664792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.664793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.664984 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.665319 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:18.665321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:18.665322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:18.665399 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:18.665864 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:18.665867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:18.665956 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.666064 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.666220 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:23:18 dist-test-slave-2rbl krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:18.688612 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.679917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.21.33.190:33599)
0929 10:23:18.680044 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.680047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.680060 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:18.680099 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.680101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.680140 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.680213 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:18.681013 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.681584 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.682419 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.682629 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.682734 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.685284 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:18.685288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:18.685290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:18.685314 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:18.686788 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.687234 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.687237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.687238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.687321 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:18.687569 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:18.687571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:18.687572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:18.687779 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:18.687885 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.688266 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.688369 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:18.691994 22762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:18.695137 22762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:18.710186 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.703624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:34117 (local address 127.0.0.1:56620)
0929 10:23:18.704325 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.704377 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.704502 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.705315 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.705321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.705345 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:18.705504 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.705511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.706950 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.706953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.708444 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.708453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.709589 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.709600 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.709719 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.709761 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.709885 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.710014 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":423,"thread_start_us":105,"threads_started":1}
I20260929 10:23:18.710208 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.704266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.21.33.130:34117)
0929 10:23:18.704683 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.704695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.704717 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:23:18.704790 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.704797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.705006 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.705169 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:18.705627 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.706839 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.708582 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.709457 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.709611 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.709758 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.709940 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.710047 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":115,"threads_started":1}
I20260929 10:23:18.710549 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.703623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.21.33.131:45213)
0929 10:23:18.704468 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.704480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.705090 (+ 610us) server_negotiation.cc:407] Connection header received
0929 10:23:18.705359 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.705364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.705549 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.705679 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:18.706003 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.706725 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.708714 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.709480 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.709574 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.709758 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.710284 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.710388 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":115,"threads_started":1}
I20260929 10:23:18.710564 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.704266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.21.33.129:37331)
0929 10:23:18.704803 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.704816 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.704834 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:23:18.704898 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.704902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.705042 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.705170 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:18.705686 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.706839 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.708829 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.709474 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.709605 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.709772 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.710333 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.710437 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":100,"threads_started":1}
I20260929 10:23:18.710232 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.704083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37331 (local address 127.0.0.1:48620)
0929 10:23:18.704649 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.704664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.704745 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.705345 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.705353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.705379 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:18.705536 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.705542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.706999 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.707002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.708714 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.708721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.709846 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.709851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.709961 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.709981 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.710053 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.710109 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":423,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260929 10:23:18.710954 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.703462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:45213 (local address 127.0.0.1:55604)
0929 10:23:18.705000 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.705013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.705171 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.705808 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.705810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.705823 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:18.705894 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.705897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.706839 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.706846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.708588 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.708598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.709836 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.709839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.709995 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.710019 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.710107 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.710797 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":75,"thread_start_us":73,"threads_started":1}
I20260929 10:23:18.713876 23040 tablet_service.cc:1467] Processing CreateTablet for tablet b99a3ba2ce774258960191649c979272 (DEFAULT_TABLE table=test-table [id=aae5e82daca34c31b20b3d82c453b881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:18.713897 23180 tablet_service.cc:1467] Processing CreateTablet for tablet b99a3ba2ce774258960191649c979272 (DEFAULT_TABLE table=test-table [id=aae5e82daca34c31b20b3d82c453b881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:18.713883 22899 tablet_service.cc:1467] Processing CreateTablet for tablet b99a3ba2ce774258960191649c979272 (DEFAULT_TABLE table=test-table [id=aae5e82daca34c31b20b3d82c453b881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:18.714855 23040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99a3ba2ce774258960191649c979272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:18.714892 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99a3ba2ce774258960191649c979272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:18.714885 22899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99a3ba2ce774258960191649c979272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:18.721738 23270 tablet_bootstrap.cc:492] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Bootstrap starting.
I20260929 10:23:18.722733 23271 tablet_bootstrap.cc:492] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Bootstrap starting.
I20260929 10:23:18.723826 23270 tablet_bootstrap.cc:654] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:18.724542 23270 log.cc:824] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:18.724843 23271 tablet_bootstrap.cc:654] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:18.725186 23272 tablet_bootstrap.cc:492] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Bootstrap starting.
I20260929 10:23:18.725852 23271 log.cc:824] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:18.726763 23270 tablet_bootstrap.cc:492] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: No bootstrap required, opened a new log
I20260929 10:23:18.726956 23270 ts_tablet_manager.cc:1397] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:18.727298 23272 tablet_bootstrap.cc:654] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:18.727843 23271 tablet_bootstrap.cc:492] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: No bootstrap required, opened a new log
I20260929 10:23:18.728030 23271 ts_tablet_manager.cc:1397] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:23:18.728116 23272 log.cc:824] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:18.729882 23272 tablet_bootstrap.cc:492] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: No bootstrap required, opened a new log
I20260929 10:23:18.730069 23272 ts_tablet_manager.cc:1397] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:18.730908 23270 raft_consensus.cc:348] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.731280 23270 raft_consensus.cc:374] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:18.731381 23270 raft_consensus.cc:729] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a6b3760be4423fb32807487abab116, State: Initialized, Role: FOLLOWER
I20260929 10:23:18.731745 23271 raft_consensus.cc:348] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.731931 23270 consensus_queue.cc:260] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.732040 23271 raft_consensus.cc:374] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:18.732105 23271 raft_consensus.cc:729] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbb2e82947a4d60b18b4cb07e002709, State: Initialized, Role: FOLLOWER
I20260929 10:23:18.732606 23106 heartbeater.cc:503] Master 127.21.33.190:33599 was elected leader, sending a full tablet report...
I20260929 10:23:18.732935 23270 ts_tablet_manager.cc:1428] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:23:18.733376 23271 consensus_queue.cc:260] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.733454 23272 raft_consensus.cc:348] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.733760 23272 raft_consensus.cc:374] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:18.733851 23272 raft_consensus.cc:729] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e618cdc16084bb985c3aa0798056d60, State: Initialized, Role: FOLLOWER
I20260929 10:23:18.734313 23272 consensus_queue.cc:260] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.734443 23271 ts_tablet_manager.cc:1428] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:23:18.734942 23247 heartbeater.cc:503] Master 127.21.33.190:33599 was elected leader, sending a full tablet report...
I20260929 10:23:18.735275 23272 ts_tablet_manager.cc:1428] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:23:18.735601 22965 heartbeater.cc:503] Master 127.21.33.190:33599 was elected leader, sending a full tablet report...
W20260929 10:23:18.902729 23248 tablet.cc:2369] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:18.932344 23277 raft_consensus.cc:482] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:18.932574 23277 raft_consensus.cc:504] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.933911 23277 leader_election.cc:288] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131:45213), 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117)
I20260929 10:23:18.939531 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.934285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:39319 (local address 127.21.33.130:34117)
0929 10:23:18.934546 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.934550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.934610 (+ 60us) server_negotiation.cc:407] Connection header received
0929 10:23:18.934800 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.934804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.934858 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.934942 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:18.935368 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.936177 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.938119 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.938759 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.938905 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.938959 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.939319 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.939397 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 10:23:18.939531 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.934314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:40863 (local address 127.21.33.131:45213)
0929 10:23:18.934448 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:18.934455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:18.934740 (+ 285us) server_negotiation.cc:407] Connection header received
0929 10:23:18.934815 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:18.934818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:18.934862 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:18.934942 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:18.935368 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.936207 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.938113 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.938785 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.938905 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.938959 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:18.939319 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 10:23:18.939395 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:23:18.939616 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.934155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:45213 (local address 127.21.33.129:40863)
0929 10:23:18.934636 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.934647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.934736 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.934972 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.934975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.935003 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:18.935259 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.935265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.936318 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.936321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.937915 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.937929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.938935 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.938940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.939135 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.939199 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.939284 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.939400 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":404,"spinlock_wait_cycles":1664,"thread_start_us":110,"threads_started":1}
I20260929 10:23:18.939644 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:18.934155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:34117 (local address 127.21.33.129:39319)
0929 10:23:18.934506 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:18.934523 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:18.934667 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:18.934979 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:18.934982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:18.935007 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:18.935243 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.935251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.936318 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.936321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.937923 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:18.937929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:18.938937 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:18.938940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:18.939132 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:18.939195 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 10:23:18.939284 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:23:18.939400 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":174,"spinlock_wait_cycles":7808,"thread_start_us":134,"threads_started":1}
I20260929 10:23:18.940451 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a3ba2ce774258960191649c979272" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e618cdc16084bb985c3aa0798056d60" is_pre_election: true
I20260929 10:23:18.940789 23193 raft_consensus.cc:2454] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 0.
I20260929 10:23:18.940769 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a3ba2ce774258960191649c979272" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a6b3760be4423fb32807487abab116" is_pre_election: true
I20260929 10:23:18.941098 23060 raft_consensus.cc:2454] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 0.
I20260929 10:23:18.941363 22851 leader_election.cc:302] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e618cdc16084bb985c3aa0798056d60, 2cbb2e82947a4d60b18b4cb07e002709; no voters:
W20260929 10:23:18.941493 23107 tablet.cc:2369] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:18.941663 23277 raft_consensus.cc:2781] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:18.941749 23277 raft_consensus.cc:482] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:18.941789 23277 raft_consensus.cc:3037] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:18.942873 23277 raft_consensus.cc:504] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.943233 23277 leader_election.cc:288] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 election: Requested vote from peers 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131:45213), 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117)
I20260929 10:23:18.943625 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a3ba2ce774258960191649c979272" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a6b3760be4423fb32807487abab116"
I20260929 10:23:18.943625 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99a3ba2ce774258960191649c979272" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e618cdc16084bb985c3aa0798056d60"
I20260929 10:23:18.943781 23060 raft_consensus.cc:3037] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:18.943779 23193 raft_consensus.cc:3037] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:18.945258 23060 raft_consensus.cc:2454] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 1.
I20260929 10:23:18.945607 22853 leader_election.cc:302] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a6b3760be4423fb32807487abab116, 2cbb2e82947a4d60b18b4cb07e002709; no voters:
I20260929 10:23:18.945662 23193 raft_consensus.cc:2454] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 1.
I20260929 10:23:18.945833 23277 raft_consensus.cc:2781] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:18.946110 23277 raft_consensus.cc:686] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 LEADER]: Becoming Leader. State: Replica: 2cbb2e82947a4d60b18b4cb07e002709, State: Running, Role: LEADER
I20260929 10:23:18.946491 23277 consensus_queue.cc:237] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } }
I20260929 10:23:18.949734 22762 catalog_manager.cc:5495] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cbb2e82947a4d60b18b4cb07e002709 (127.21.33.129). New cstate: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } health_report { overall_health: HEALTHY } } }
W20260929 10:23:18.960470 22966 tablet.cc:2369] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:19.013340 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.009399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.21.33.129:37331)
0929 10:23:19.009554 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.009557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.009573 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:19.009787 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.009790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.009844 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.009928 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:19.010251 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.010755 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.011668 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.011892 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.011976 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.012357 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.012866 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.012951 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.013073 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.013127 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:23:19.019387 23060 raft_consensus.cc:1261] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Refusing update from remote peer 2cbb2e82947a4d60b18b4cb07e002709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:19.019421 23193 raft_consensus.cc:1261] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Refusing update from remote peer 2cbb2e82947a4d60b18b4cb07e002709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:19.020152 23277 consensus_queue.cc:1035] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:19.020422 23281 consensus_queue.cc:1035] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:19.039499 22762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:19.039688 22762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:19.042099 22762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76: Sending DeleteTablet for 3 replicas of tablet b99a3ba2ce774258960191649c979272
I20260929 10:23:19.042878 23040 tablet_service.cc:1514] Processing DeleteTablet for tablet b99a3ba2ce774258960191649c979272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:19 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:56620
I20260929 10:23:19.042891 23180 tablet_service.cc:1514] Processing DeleteTablet for tablet b99a3ba2ce774258960191649c979272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:19 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:55604
I20260929 10:23:19.043022 22899 tablet_service.cc:1514] Processing DeleteTablet for tablet b99a3ba2ce774258960191649c979272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:19 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:48620
I20260929 10:23:19.043731 23294 tablet_replica.cc:330] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: stopping tablet replica
I20260929 10:23:19.043963 22762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:19.044384 22762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:19.045336 23294 raft_consensus.cc:2229] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:19.045660 23294 raft_consensus.cc:2258] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:19.046182 23295 tablet_replica.cc:330] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: stopping tablet replica
I20260929 10:23:19.046446 23295 raft_consensus.cc:2229] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:19.046797 23295 raft_consensus.cc:2258] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:19.048297 23294 ts_tablet_manager.cc:1905] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:19.049105 22899 tablet_service.cc:1467] Processing CreateTablet for tablet 42555b06797849cb9dfc0727616144c4 (DEFAULT_TABLE table=test-table [id=c42fa126bcd64e4d9ba5c9a039a8ac18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:19.049445 22899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42555b06797849cb9dfc0727616144c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:19.049980 23180 tablet_service.cc:1467] Processing CreateTablet for tablet 42555b06797849cb9dfc0727616144c4 (DEFAULT_TABLE table=test-table [id=c42fa126bcd64e4d9ba5c9a039a8ac18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:19.050201 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42555b06797849cb9dfc0727616144c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:19.051030 23294 ts_tablet_manager.cc:1918] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:19.051123 23294 log.cc:1196] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/b99a3ba2ce774258960191649c979272
I20260929 10:23:19.051472 23294 ts_tablet_manager.cc:1939] T b99a3ba2ce774258960191649c979272 P 0e618cdc16084bb985c3aa0798056d60: Deleting consensus metadata
I20260929 10:23:19.051924 23271 tablet_bootstrap.cc:492] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: Bootstrap starting.
I20260929 10:23:19.052395 23272 tablet_bootstrap.cc:492] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: Bootstrap starting.
I20260929 10:23:19.052884 23271 tablet_bootstrap.cc:654] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:19.052970 23293 tablet_replica.cc:330] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: stopping tablet replica
I20260929 10:23:19.053025 22747 catalog_manager.cc:4841] TS 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131:45213): tablet b99a3ba2ce774258960191649c979272 (table test-table [id=aae5e82daca34c31b20b3d82c453b881]) successfully deleted
I20260929 10:23:19.053341 23272 tablet_bootstrap.cc:654] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:19.053351 23293 raft_consensus.cc:2229] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:19.053680 23293 raft_consensus.cc:2258] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:19.054437 23040 tablet_service.cc:1467] Processing CreateTablet for tablet 42555b06797849cb9dfc0727616144c4 (DEFAULT_TABLE table=test-table [id=c42fa126bcd64e4d9ba5c9a039a8ac18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:19.054490 23295 ts_tablet_manager.cc:1905] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:19.054706 23040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42555b06797849cb9dfc0727616144c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:19.056924 23270 tablet_bootstrap.cc:492] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: Bootstrap starting.
I20260929 10:23:19.056914 23295 ts_tablet_manager.cc:1918] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:19.057003 23295 log.cc:1196] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/b99a3ba2ce774258960191649c979272
I20260929 10:23:19.057330 23295 ts_tablet_manager.cc:1939] T b99a3ba2ce774258960191649c979272 P 2cbb2e82947a4d60b18b4cb07e002709: Deleting consensus metadata
I20260929 10:23:19.057899 23272 tablet_bootstrap.cc:492] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: No bootstrap required, opened a new log
I20260929 10:23:19.057926 23270 tablet_bootstrap.cc:654] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:19.057981 23272 ts_tablet_manager.cc:1397] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260929 10:23:19.058487 23272 raft_consensus.cc:348] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.058509 22748 catalog_manager.cc:4841] TS 2cbb2e82947a4d60b18b4cb07e002709 (127.21.33.129:37331): tablet b99a3ba2ce774258960191649c979272 (table test-table [id=aae5e82daca34c31b20b3d82c453b881]) successfully deleted
I20260929 10:23:19.058600 23272 raft_consensus.cc:374] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:19.058632 23272 raft_consensus.cc:729] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e618cdc16084bb985c3aa0798056d60, State: Initialized, Role: FOLLOWER
I20260929 10:23:19.058754 23272 consensus_queue.cc:260] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.059039 23272 ts_tablet_manager.cc:1428] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:23:19.060091 23293 ts_tablet_manager.cc:1905] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:19.060187 23270 tablet_bootstrap.cc:492] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: No bootstrap required, opened a new log
I20260929 10:23:19.060249 23270 ts_tablet_manager.cc:1397] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:19.060928 23270 raft_consensus.cc:348] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.061034 23270 raft_consensus.cc:374] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:19.061066 23270 raft_consensus.cc:729] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a6b3760be4423fb32807487abab116, State: Initialized, Role: FOLLOWER
I20260929 10:23:19.061190 23270 consensus_queue.cc:260] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.061491 23270 ts_tablet_manager.cc:1428] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:19.062158 23271 tablet_bootstrap.cc:492] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: No bootstrap required, opened a new log
I20260929 10:23:19.062240 23271 ts_tablet_manager.cc:1397] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260929 10:23:19.062621 23293 ts_tablet_manager.cc:1918] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:19.062711 23293 log.cc:1196] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/b99a3ba2ce774258960191649c979272
I20260929 10:23:19.062678 23271 raft_consensus.cc:348] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.062775 23271 raft_consensus.cc:374] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:19.062805 23271 raft_consensus.cc:729] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbb2e82947a4d60b18b4cb07e002709, State: Initialized, Role: FOLLOWER
I20260929 10:23:19.062912 23271 consensus_queue.cc:260] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.063033 23293 ts_tablet_manager.cc:1939] T b99a3ba2ce774258960191649c979272 P 08a6b3760be4423fb32807487abab116: Deleting consensus metadata
I20260929 10:23:19.063130 23271 ts_tablet_manager.cc:1428] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:19.064184 22749 catalog_manager.cc:4841] TS 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117): tablet b99a3ba2ce774258960191649c979272 (table test-table [id=aae5e82daca34c31b20b3d82c453b881]) successfully deleted
I20260929 10:23:19.142421 23278 raft_consensus.cc:482] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:19.142596 23278 raft_consensus.cc:504] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.143890 23278 leader_election.cc:288] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cbb2e82947a4d60b18b4cb07e002709 (127.21.33.129:37331), 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117)
I20260929 10:23:19.148538 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.144140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37331 (local address 127.21.33.131:45595)
0929 10:23:19.144264 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:19.144286 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:19.144387 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:19.144875 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:19.144879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:19.144908 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:19.145029 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.145035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.146036 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.146039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.147250 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.147260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.148217 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.148220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.148327 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.148344 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 10:23:19.148404 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 10:23:19.148461 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 10:23:19.148553 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.144523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:52457 (local address 127.21.33.130:34117)
0929 10:23:19.144659 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.144664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.144794 (+ 130us) server_negotiation.cc:407] Connection header received
0929 10:23:19.144971 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.144974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.145014 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.145051 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:19.145499 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.146193 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.147539 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.147944 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.148085 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.148139 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.148338 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.148420 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:23:19.148541 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.144140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:34117 (local address 127.21.33.131:52457)
0929 10:23:19.144694 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:19.144707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:19.144808 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:19.145225 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:19.145228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:19.145245 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:19.145364 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.145371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.146313 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.146316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.147430 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.147437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.148086 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.148089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.148196 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.148221 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:23:19.148297 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:23:19.148371 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":45,"thread_start_us":88,"threads_started":1}
I20260929 10:23:19.148833 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.144483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:45595 (local address 127.21.33.129:37331)
0929 10:23:19.144600 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.144605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.144621 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:19.144688 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.144691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.144753 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.144827 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:19.145181 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.145914 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.147403 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.148021 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.148148 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.148198 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.148605 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.148703 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:23:19.149379 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42555b06797849cb9dfc0727616144c4" candidate_uuid: "0e618cdc16084bb985c3aa0798056d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a6b3760be4423fb32807487abab116" is_pre_election: true
I20260929 10:23:19.149363 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42555b06797849cb9dfc0727616144c4" candidate_uuid: "0e618cdc16084bb985c3aa0798056d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbb2e82947a4d60b18b4cb07e002709" is_pre_election: true
I20260929 10:23:19.149544 23060 raft_consensus.cc:2454] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e618cdc16084bb985c3aa0798056d60 in term 0.
I20260929 10:23:19.149595 22919 raft_consensus.cc:2454] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e618cdc16084bb985c3aa0798056d60 in term 0.
I20260929 10:23:19.149888 23134 leader_election.cc:302] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a6b3760be4423fb32807487abab116, 0e618cdc16084bb985c3aa0798056d60; no voters:
I20260929 10:23:19.150172 23278 raft_consensus.cc:2781] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:19.150233 23278 raft_consensus.cc:482] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:19.150271 23278 raft_consensus.cc:3037] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:19.151304 23278 raft_consensus.cc:504] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.151687 23278 leader_election.cc:288] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [CANDIDATE]: Term 1 election: Requested vote from peers 2cbb2e82947a4d60b18b4cb07e002709 (127.21.33.129:37331), 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117)
I20260929 10:23:19.152032 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42555b06797849cb9dfc0727616144c4" candidate_uuid: "0e618cdc16084bb985c3aa0798056d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbb2e82947a4d60b18b4cb07e002709"
I20260929 10:23:19.152030 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42555b06797849cb9dfc0727616144c4" candidate_uuid: "0e618cdc16084bb985c3aa0798056d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a6b3760be4423fb32807487abab116"
I20260929 10:23:19.152153 22919 raft_consensus.cc:3037] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:19.152164 23060 raft_consensus.cc:3037] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:19.153348 23060 raft_consensus.cc:2454] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e618cdc16084bb985c3aa0798056d60 in term 1.
I20260929 10:23:19.153404 22919 raft_consensus.cc:2454] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e618cdc16084bb985c3aa0798056d60 in term 1.
I20260929 10:23:19.153723 23133 leader_election.cc:302] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e618cdc16084bb985c3aa0798056d60, 2cbb2e82947a4d60b18b4cb07e002709; no voters:
I20260929 10:23:19.153889 23278 raft_consensus.cc:2781] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:19.154072 23278 raft_consensus.cc:686] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 1 LEADER]: Becoming Leader. State: Replica: 0e618cdc16084bb985c3aa0798056d60, State: Running, Role: LEADER
I20260929 10:23:19.154310 23278 consensus_queue.cc:237] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.156786 22761 catalog_manager.cc:5495] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131). New cstate: current_term: 1 leader_uuid: "0e618cdc16084bb985c3aa0798056d60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } health_report { overall_health: HEALTHY } } }
Sep 29 10:23:19 dist-test-slave-2rbl krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677397, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:19.220664 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.210946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.21.33.190:33599)
0929 10:23:19.211207 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.211215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.211295 (+ 80us) server_negotiation.cc:407] Connection header received
0929 10:23:19.211420 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.211423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.211471 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.211568 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:19.212269 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.213102 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.214008 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.214386 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.214482 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.216936 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:19.216939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:19.216942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:19.216965 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:19.218648 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:19.219102 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:19.219106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:19.219108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:19.219190 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:19.219639 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:19.219642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:19.219644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:19.219852 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:19.219942 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.220263 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.220452 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
I20260929 10:23:19.220700 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.210884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33599 (local address 127.0.0.1:49804)
0929 10:23:19.211179 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:19.211207 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:19.211324 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:19.211695 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:19.211698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:19.212047 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:19.212137 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.212143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.213254 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.213258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.213898 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.213907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.214518 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.214521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.214590 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.215178 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:19.215190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:19.216796 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:19.218757 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:19.218765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:19.218770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:19.219011 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:19.219378 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:19.219384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:19.219391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:19.219488 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:19.219965 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:19.219971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:19.220088 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 10:23:19.220198 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 10:23:19.220357 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260929 10:23:19.225412 22761 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:49804:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:23:19.226359 22761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:19.231163 22899 tablet_service.cc:1467] Processing CreateTablet for tablet c0deb5e4589a444391eab22587dcaf44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=95eb496bc16c40dcad226e242e107c15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:19.231184 23040 tablet_service.cc:1467] Processing CreateTablet for tablet c0deb5e4589a444391eab22587dcaf44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=95eb496bc16c40dcad226e242e107c15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:19.231272 23180 tablet_service.cc:1467] Processing CreateTablet for tablet c0deb5e4589a444391eab22587dcaf44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=95eb496bc16c40dcad226e242e107c15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:19.231570 22899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0deb5e4589a444391eab22587dcaf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:19.231571 23040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0deb5e4589a444391eab22587dcaf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:19.231688 23180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0deb5e4589a444391eab22587dcaf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:19.234359 23271 tablet_bootstrap.cc:492] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: Bootstrap starting.
I20260929 10:23:19.234378 23272 tablet_bootstrap.cc:492] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: Bootstrap starting.
I20260929 10:23:19.234650 23270 tablet_bootstrap.cc:492] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: Bootstrap starting.
I20260929 10:23:19.235229 23271 tablet_bootstrap.cc:654] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:19.235229 23272 tablet_bootstrap.cc:654] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:19.235528 23270 tablet_bootstrap.cc:654] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:19.236397 23271 tablet_bootstrap.cc:492] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: No bootstrap required, opened a new log
I20260929 10:23:19.236487 23271 ts_tablet_manager.cc:1397] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:23:19.236613 23272 tablet_bootstrap.cc:492] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: No bootstrap required, opened a new log
I20260929 10:23:19.236702 23272 ts_tablet_manager.cc:1397] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:23:19.236924 23271 raft_consensus.cc:348] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.237010 23271 raft_consensus.cc:374] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:19.237031 23271 raft_consensus.cc:729] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbb2e82947a4d60b18b4cb07e002709, State: Initialized, Role: FOLLOWER
I20260929 10:23:19.237123 23271 consensus_queue.cc:260] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.237071 23272 raft_consensus.cc:348] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.237196 23272 raft_consensus.cc:374] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:19.237258 23272 raft_consensus.cc:729] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e618cdc16084bb985c3aa0798056d60, State: Initialized, Role: FOLLOWER
I20260929 10:23:19.237371 23272 consensus_queue.cc:260] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.237423 23271 ts_tablet_manager.cc:1428] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:19.237375 23277 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.237500 23277 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:19.237638 23272 ts_tablet_manager.cc:1428] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:19.237903 23278 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.238047 23278 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:19.238576 23270 tablet_bootstrap.cc:492] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: No bootstrap required, opened a new log
I20260929 10:23:19.238698 23270 ts_tablet_manager.cc:1397] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 10:23:19.239164 23270 raft_consensus.cc:348] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.239285 23270 raft_consensus.cc:374] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:19.239321 23270 raft_consensus.cc:729] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a6b3760be4423fb32807487abab116, State: Initialized, Role: FOLLOWER
I20260929 10:23:19.239444 23270 consensus_queue.cc:260] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.239728 23270 ts_tablet_manager.cc:1428] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:19.239938 23276 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.240067 23276 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:19.257467 23277 raft_consensus.cc:482] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:19.257651 23277 raft_consensus.cc:504] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.258100 23277 leader_election.cc:288] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117), 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131:45213)
I20260929 10:23:19.258545 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0deb5e4589a444391eab22587dcaf44" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e618cdc16084bb985c3aa0798056d60" is_pre_election: true
I20260929 10:23:19.258617 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0deb5e4589a444391eab22587dcaf44" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a6b3760be4423fb32807487abab116" is_pre_election: true
I20260929 10:23:19.258718 23193 raft_consensus.cc:2454] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 0.
I20260929 10:23:19.258759 23060 raft_consensus.cc:2454] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 0.
I20260929 10:23:19.259087 22853 leader_election.cc:302] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08a6b3760be4423fb32807487abab116, 0e618cdc16084bb985c3aa0798056d60, 2cbb2e82947a4d60b18b4cb07e002709; no voters:
I20260929 10:23:19.259347 23277 raft_consensus.cc:2781] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:19.259430 23277 raft_consensus.cc:482] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:19.259485 23277 raft_consensus.cc:3037] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:19.260496 23277 raft_consensus.cc:504] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.260839 23277 leader_election.cc:288] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 election: Requested vote from peers 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117), 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131:45213)
I20260929 10:23:19.261320 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0deb5e4589a444391eab22587dcaf44" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e618cdc16084bb985c3aa0798056d60"
I20260929 10:23:19.261309 23060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0deb5e4589a444391eab22587dcaf44" candidate_uuid: "2cbb2e82947a4d60b18b4cb07e002709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a6b3760be4423fb32807487abab116"
I20260929 10:23:19.261461 23193 raft_consensus.cc:3037] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:19.261471 23060 raft_consensus.cc:3037] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:19.262400 23193 raft_consensus.cc:2454] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 1.
I20260929 10:23:19.262413 23060 raft_consensus.cc:2454] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbb2e82947a4d60b18b4cb07e002709 in term 1.
I20260929 10:23:19.262734 22851 leader_election.cc:302] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08a6b3760be4423fb32807487abab116, 0e618cdc16084bb985c3aa0798056d60, 2cbb2e82947a4d60b18b4cb07e002709; no voters:
I20260929 10:23:19.262951 23277 raft_consensus.cc:2781] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:19.263064 23277 raft_consensus.cc:686] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 LEADER]: Becoming Leader. State: Replica: 2cbb2e82947a4d60b18b4cb07e002709, State: Running, Role: LEADER
I20260929 10:23:19.263193 23277 consensus_queue.cc:237] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } }
I20260929 10:23:19.263875 23281 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: TxnStatusTablet state changed. Reason: New leader 2cbb2e82947a4d60b18b4cb07e002709. Latest consensus state: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.263969 23281 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:19.264261 23313 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:23:19.264429 23313 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:23:19.264940 22762 catalog_manager.cc:5495] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cbb2e82947a4d60b18b4cb07e002709 (127.21.33.129). New cstate: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:19.282449 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.278675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.21.33.129:37331)
0929 10:23:19.278790 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.278794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.278810 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:19.278995 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.278999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.279049 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.279157 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:19.279436 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.280240 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.281131 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.281510 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.281618 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.281991 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.282074 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.282145 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.282263 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.282319 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:23:19.282416 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.278571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37331 (local address 127.0.0.1:48650)
0929 10:23:19.278714 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:19.278733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:19.278843 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:19.279205 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:19.279210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:19.279218 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:19.279329 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.279335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.280369 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.280372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.281003 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.281011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.281680 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.281684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.281765 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.281841 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.282164 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.282176 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 10:23:19.282231 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 10:23:19.282303 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:23:19.284865 23060 raft_consensus.cc:1261] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Refusing update from remote peer 2cbb2e82947a4d60b18b4cb07e002709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:19.284963 23193 raft_consensus.cc:1261] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Refusing update from remote peer 2cbb2e82947a4d60b18b4cb07e002709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:19.285293 23281 consensus_queue.cc:1035] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:19.285420 23277 consensus_queue.cc:1035] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:19.287009 23276 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: TxnStatusTablet state changed. Reason: New leader 2cbb2e82947a4d60b18b4cb07e002709. Latest consensus state: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.287143 23276 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:19.287829 23278 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: TxnStatusTablet state changed. Reason: New leader 2cbb2e82947a4d60b18b4cb07e002709. Latest consensus state: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.287941 23278 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:19.289740 23281 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.289867 23281 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:19.289750 23277 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.290452 23277 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 2cbb2e82947a4d60b18b4cb07e002709: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:19.290341 23284 mvcc.cc:205] Tried to move back new op lower bound from 7334614627466874880 to 7334614627383361536. Current Snapshot: MvccSnapshot[applied={T|T < 7334614627466874880 or (T in {7334614627466874880})}]
I20260929 10:23:19.291035 23278 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.291163 23278 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 0e618cdc16084bb985c3aa0798056d60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:19.291347 23285 mvcc.cc:205] Tried to move back new op lower bound from 7334614627466874880 to 7334614627383361536. Current Snapshot: MvccSnapshot[applied={T|T < 7334614627466874880 or (T in {7334614627466874880})}]
I20260929 10:23:19.291860 23286 mvcc.cc:205] Tried to move back new op lower bound from 7334614627466874880 to 7334614627383361536. Current Snapshot: MvccSnapshot[applied={T|T < 7334614627466874880 or (T in {7334614627466874880})}]
I20260929 10:23:19.292132 23276 tablet_replica.cc:439] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cbb2e82947a4d60b18b4cb07e002709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 } } peers { permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 } } peers { permanent_uuid: "0e618cdc16084bb985c3aa0798056d60" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45213 } } }
I20260929 10:23:19.292218 23276 tablet_replica.cc:442] T c0deb5e4589a444391eab22587dcaf44 P 08a6b3760be4423fb32807487abab116: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:19.298748 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.294529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.21.33.131:45213)
0929 10:23:19.294655 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.294660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.294702 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:23:19.294908 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.294911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.294972 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.295070 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:19.295390 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.296151 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.297061 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.297442 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.297544 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.297814 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.298362 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.298476 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.298549 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.298604 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:23:19.309738 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.305795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:58173 (local address 127.21.33.129:37331)
0929 10:23:19.305932 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.305938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.306168 (+ 230us) server_negotiation.cc:407] Connection header received
0929 10:23:19.306246 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.306248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.306293 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.306359 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:19.307005 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.307474 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.308696 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.308897 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.308975 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.309340 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.309409 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.309475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.309592 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.309639 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:23:19.309804 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.305695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37331 (local address 127.21.33.131:58173)
0929 10:23:19.306036 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:19.306052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:19.306164 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:19.306390 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:19.306393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:19.306401 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:19.306892 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.306899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.307581 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.307585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.308571 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.308581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.309017 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.309021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.309122 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.309173 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.309499 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.309505 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:23:19.309570 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:23:19.309647 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":130,"threads_started":1}
I20260929 10:23:19.316442 23060 raft_consensus.cc:1261] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Refusing update from remote peer 0e618cdc16084bb985c3aa0798056d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:19.316461 22919 raft_consensus.cc:1261] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Refusing update from remote peer 0e618cdc16084bb985c3aa0798056d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:19.316888 23278 consensus_queue.cc:1035] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a6b3760be4423fb32807487abab116" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 34117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:19.317058 23300 consensus_queue.cc:1035] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbb2e82947a4d60b18b4cb07e002709" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:19.339138 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.335399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:45213 (local address 127.21.33.129:37043)
0929 10:23:19.335731 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:19.335748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:19.335840 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:19.336063 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:19.336066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:19.336074 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:19.336191 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.336197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.337031 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.337035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.337951 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:19.337961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.338425 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.338428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:19.338513 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.338564 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.338859 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.338867 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:23:19.338928 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 10:23:19.339008 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260929 10:23:19.339129 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.335734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:37043 (local address 127.21.33.131:45213)
0929 10:23:19.335863 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.335867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.335882 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:23:19.335930 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.335932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.335971 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.336034 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:19.336312 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.336917 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.338070 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.338315 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.338423 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.338695 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:19.338772 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:19.338855 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.338947 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.338994 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:23:19.366556 22762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:19.366751 22762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:19.369026 22762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c471b20f38a4933936d80ac5869ad76: Sending DeleteTablet for 3 replicas of tablet 42555b06797849cb9dfc0727616144c4
I20260929 10:23:19.369589 22899 tablet_service.cc:1514] Processing DeleteTablet for tablet 42555b06797849cb9dfc0727616144c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:19 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:48620
I20260929 10:23:19.369663 23040 tablet_service.cc:1514] Processing DeleteTablet for tablet 42555b06797849cb9dfc0727616144c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:19 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:56620
I20260929 10:23:19.369769 23295 tablet_replica.cc:330] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: stopping tablet replica
I20260929 10:23:19.369845 23293 tablet_replica.cc:330] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: stopping tablet replica
I20260929 10:23:19.369879 23295 raft_consensus.cc:2229] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:19.369951 23293 raft_consensus.cc:2229] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:19.369987 23295 raft_consensus.cc:2258] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:19.370064 23293 raft_consensus.cc:2258] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:19.370678 23180 tablet_service.cc:1514] Processing DeleteTablet for tablet 42555b06797849cb9dfc0727616144c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:19 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:55604
I20260929 10:23:19.370851 23294 tablet_replica.cc:330] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: stopping tablet replica
I20260929 10:23:19.370942 23295 ts_tablet_manager.cc:1905] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:19.370954 23294 raft_consensus.cc:2229] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:19.371255 23294 raft_consensus.cc:2258] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:19.371989 23294 ts_tablet_manager.cc:1905] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:19.374142 23295 ts_tablet_manager.cc:1918] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:19.374302 23295 log.cc:1196] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/42555b06797849cb9dfc0727616144c4
I20260929 10:23:19.374639 23295 ts_tablet_manager.cc:1939] T 42555b06797849cb9dfc0727616144c4 P 2cbb2e82947a4d60b18b4cb07e002709: Deleting consensus metadata
I20260929 10:23:19.374742 23294 ts_tablet_manager.cc:1918] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:19.374827 23294 log.cc:1196] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/42555b06797849cb9dfc0727616144c4
I20260929 10:23:19.375088 23294 ts_tablet_manager.cc:1939] T 42555b06797849cb9dfc0727616144c4 P 0e618cdc16084bb985c3aa0798056d60: Deleting consensus metadata
I20260929 10:23:19.375564 22748 catalog_manager.cc:4841] TS 2cbb2e82947a4d60b18b4cb07e002709 (127.21.33.129:37331): tablet 42555b06797849cb9dfc0727616144c4 (table test-table [id=c42fa126bcd64e4d9ba5c9a039a8ac18]) successfully deleted
I20260929 10:23:19.376144 23293 ts_tablet_manager.cc:1905] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:19.378455 22747 catalog_manager.cc:4841] TS 0e618cdc16084bb985c3aa0798056d60 (127.21.33.131:45213): tablet 42555b06797849cb9dfc0727616144c4 (table test-table [id=c42fa126bcd64e4d9ba5c9a039a8ac18]) successfully deleted
I20260929 10:23:19.378911 23293 ts_tablet_manager.cc:1918] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:19.378988 23293 log.cc:1196] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/42555b06797849cb9dfc0727616144c4
I20260929 10:23:19.379228 23293 ts_tablet_manager.cc:1939] T 42555b06797849cb9dfc0727616144c4 P 08a6b3760be4423fb32807487abab116: Deleting consensus metadata
I20260929 10:23:19.380368 22749 catalog_manager.cc:4841] TS 08a6b3760be4423fb32807487abab116 (127.21.33.130:34117): tablet 42555b06797849cb9dfc0727616144c4 (table test-table [id=c42fa126bcd64e4d9ba5c9a039a8ac18]) successfully deleted
Sep 29 10:23:19 dist-test-slave-2rbl krb5kdc[22717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:19.392768 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.380527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.21.33.129:37331)
0929 10:23:19.380639 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.380654 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.381142 (+ 488us) server_negotiation.cc:407] Connection header received
0929 10:23:19.381303 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.381306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.381357 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.381438 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:19.382250 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.383060 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.384216 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.384573 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.384690 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.388656 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:19.388670 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:19.388688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:19.388719 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:19.390823 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:19.391349 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:19.391356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:19.391358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:19.391451 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:19.391740 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:19.391743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:19.391745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:19.392043 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:19.392258 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.392474 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.392603 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 10:23:19.393428 22959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48662
I20260929 10:23:19.405145 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:19.395314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.21.33.190:33599)
0929 10:23:19.395452 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:19.395456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:19.395689 (+ 233us) server_negotiation.cc:407] Connection header received
0929 10:23:19.395789 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:19.395793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:19.395849 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:19.395939 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:19.396726 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.397589 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.398749 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:19.399091 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:19.399222 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:19.400799 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:19.400804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:19.400806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:19.400835 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:19.402620 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:19.403131 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:19.403135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:19.403137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:19.403244 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:19.403570 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:19.403574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:19.403576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:19.404415 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:19.404565 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:19.404840 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 10:23:19.404973 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 10:23:19.405570 22762 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49818
I20260929 10:23:19.407132 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22834
I20260929 10:23:19.417461 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22974
I20260929 10:23:19.423295 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 23115
I20260929 10:23:19.428258 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 22730
2026-09-29T10:23:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2872 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:19 dist-test-slave-2rbl krb5kdc[23333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:19 dist-test-slave-2rbl krb5kdc[23333](info): set up 2 sockets
Sep 29 10:23:19 dist-test-slave-2rbl krb5kdc[23333](info): commencing operation
krb5kdc: starting...
W20260929 10:23:20.477861 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:20 dist-test-slave-2rbl krb5kdc[23333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:20.605288 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:41943
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:41943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:20.715961 23346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:20.716271 23346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:20.716356 23346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:20.720456 23346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:20.720538 23346 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:23:20.720570 23346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:20.720643 23346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:20.720681 23346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:20.720712 23346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:20.726509 23346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:41943
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:41943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:20.727546 23346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:20.728603 23346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:20.737182 23354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:20.737176 23351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:20.737183 23352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:20.737524 23346 server_base.cc:1034] running on GCE node
I20260929 10:23:20.738123 23346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:20.739194 23346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:20.740372 23346 hybrid_clock.cc:648] HybridClock initialized: now 1790677400740357 us; error 27 us; skew 500 ppm
Sep 29 10:23:20 dist-test-slave-2rbl krb5kdc[23333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677400, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:20.743281 23346 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:20.744244 23346 webserver.cc:460] Webserver started at http://127.21.33.190:35155/ using document root <none> and password file <none>
I20260929 10:23:20.744817 23346 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:20.744913 23346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:20.745148 23346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:20.747187 23346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "4f770622d8e846b4b7d8cd7af5268fe3"
format_stamp: "Formatted at 2026-09-29 10:23:20 on dist-test-slave-2rbl"
server_key: "f46d863177371aeab083480923066a6c"
server_key_iv: "e1d3970b0df3fb619f66a20312af35f9"
server_key_version: "encryptionkey@0"
I20260929 10:23:20.747756 23346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "4f770622d8e846b4b7d8cd7af5268fe3"
format_stamp: "Formatted at 2026-09-29 10:23:20 on dist-test-slave-2rbl"
server_key: "f46d863177371aeab083480923066a6c"
server_key_iv: "e1d3970b0df3fb619f66a20312af35f9"
server_key_version: "encryptionkey@0"
I20260929 10:23:20.751593 23346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 10:23:20.754057 23361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:20.755316 23346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:20.755470 23346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "4f770622d8e846b4b7d8cd7af5268fe3"
format_stamp: "Formatted at 2026-09-29 10:23:20 on dist-test-slave-2rbl"
server_key: "f46d863177371aeab083480923066a6c"
server_key_iv: "e1d3970b0df3fb619f66a20312af35f9"
server_key_version: "encryptionkey@0"
I20260929 10:23:20.755599 23346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:20.769035 23346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:20.771993 23346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:20.772193 23346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:20.780097 23346 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:41943
I20260929 10:23:20.780124 23423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:41943 every 8 connection(s)
I20260929 10:23:20.781001 23346 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:20.781554 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 23346
I20260929 10:23:20.781689 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:20.781939 21638 external_mini_cluster.cc:1428] Setting key de47ac1b5d1d30c09aa96223092c4046
I20260929 10:23:20.784226 23424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:20.789812 23424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Bootstrap starting.
Sep 29 10:23:20 dist-test-slave-2rbl krb5kdc[23333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:20.792582 23424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:20.793962 23424 log.cc:824] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:20.795861 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:20.783290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.21.33.190:41943)
0929 10:23:20.783759 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:20.783773 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:20.783808 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:23:20.784474 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:20.784491 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:20.784808 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:20.785136 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:20.785944 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:20.786920 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:20.787761 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:20.788220 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:20.788391 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:20.791187 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:20.791192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:20.791203 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:20.791229 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:20.792952 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:20.793522 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:20.793526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:20.793530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:20.793650 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:20.793968 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:20.793970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:20.793971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:20.794417 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:20.794638 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:20.794881 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 10:23:20.795172 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":179,"threads_started":1}
I20260929 10:23:20.796025 23424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: No bootstrap required, opened a new log
I20260929 10:23:20.798979 23424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41943 } }
I20260929 10:23:20.799204 23424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:20.799304 23424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f770622d8e846b4b7d8cd7af5268fe3, State: Initialized, Role: FOLLOWER
I20260929 10:23:20.799734 23424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41943 } }
I20260929 10:23:20.799860 23424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:20.799949 23424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:20.800089 23424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:20.800988 23424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41943 } }
I20260929 10:23:20.801357 23424 leader_election.cc:302] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f770622d8e846b4b7d8cd7af5268fe3; no voters:
I20260929 10:23:20.801724 23424 leader_election.cc:288] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:20.801822 23429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:20.802062 23429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 4f770622d8e846b4b7d8cd7af5268fe3, State: Running, Role: LEADER
I20260929 10:23:20.802359 23429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41943 } }
I20260929 10:23:20.802909 23424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:20.804010 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f770622d8e846b4b7d8cd7af5268fe3. Latest consensus state: current_term: 1 leader_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41943 } } }
I20260929 10:23:20.804021 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f770622d8e846b4b7d8cd7af5268fe3" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41943 } } }
I20260929 10:23:20.804153 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:20.804153 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:20.804527 23438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:20.807538 23438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:20.813480 23438 catalog_manager.cc:1347] Generated new cluster ID: 93963c4617554490aa39f3c687df5406
I20260929 10:23:20.813557 23438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:20.824205 23438 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:20.825528 23438 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:20.839681 23438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Generated new TSK 0
I20260929 10:23:20.840430 23438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:20.876040 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:41943
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:20.985932 23450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:20.986171 23450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:20.986217 23450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:20.990046 23450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:20.990103 23450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:20.990142 23450 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:23:20.990156 23450 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:20.990188 23450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:20.995296 23450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:20.996258 23450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:20.997225 23450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:21.006970 23458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:21.007004 23456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:21.007004 23455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:21.007280 23450 server_base.cc:1034] running on GCE node
I20260929 10:23:21.007721 23450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:21.008389 23450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:21.009577 23450 hybrid_clock.cc:648] HybridClock initialized: now 1790677401009556 us; error 60 us; skew 500 ppm
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:21.012348 23450 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:21.013234 23450 webserver.cc:460] Webserver started at http://127.21.33.129:38257/ using document root <none> and password file <none>
I20260929 10:23:21.013811 23450 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:21.013901 23450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:21.014137 23450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:21.016139 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "6fc77d2670ee44f1beac22276b9aef5d"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "756846b641ef98d1e1e9efaa2609c75b"
server_key_iv: "1ec04f4b20d4c524fc11189b9bc12aeb"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.016708 23450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "6fc77d2670ee44f1beac22276b9aef5d"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "756846b641ef98d1e1e9efaa2609c75b"
server_key_iv: "1ec04f4b20d4c524fc11189b9bc12aeb"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.020431 23450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:23:21.022886 23465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:21.024222 23450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:21.024356 23450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "6fc77d2670ee44f1beac22276b9aef5d"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "756846b641ef98d1e1e9efaa2609c75b"
server_key_iv: "1ec04f4b20d4c524fc11189b9bc12aeb"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.024474 23450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:21.040699 23450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:21.043720 23450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:21.043923 23450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:21.045349 23450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:21.045419 23450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:21.045485 23450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:21.045538 23450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:21.056301 23450 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:42233
I20260929 10:23:21.056414 23581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:42233 every 8 connection(s)
I20260929 10:23:21.057443 23450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:21.059441 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.046203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:54257 (local address 127.21.33.190:41943)
0929 10:23:21.046363 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.046367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.047273 (+ 906us) server_negotiation.cc:407] Connection header received
0929 10:23:21.048502 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.048505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.048575 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.048654 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.049918 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.050446 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.051623 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.052013 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.052130 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.055371 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.055374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.055377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.055402 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.057165 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.057782 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.057786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.057788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.057889 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.058291 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.058294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.058295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.058523 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.058634 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.059112 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.059250 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:23:21.060163 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.046644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41943 (local address 127.21.33.129:54257)
0929 10:23:21.047127 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.047178 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.048239 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.048849 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.048856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.049306 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:21.049766 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.049779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.050596 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.050599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.051490 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.051499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.052180 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.052188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.052425 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.053161 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:21.053182 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:21.055182 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:21.057319 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.057323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.057334 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.057679 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.058006 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.058009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.058011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.058182 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.058639 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:21.058648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:21.058882 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.059071 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.059379 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
I20260929 10:23:21.061470 23582 heartbeater.cc:347] Connected to a master server at 127.21.33.190:41943
I20260929 10:23:21.061632 23582 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:21.062160 23582 heartbeater.cc:511] Master 127.21.33.190:41943 requested a full tablet report, sending...
I20260929 10:23:21.062323 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 23450
I20260929 10:23:21.062434 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:21.062743 21638 external_mini_cluster.cc:1428] Setting key 5f426c9c6bc5b2fbcbc3c5800c23ed71
I20260929 10:23:21.064086 23378 ts_manager.cc:194] Registered new tserver with Master: 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233)
I20260929 10:23:21.065701 23378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:54257
I20260929 10:23:21.071913 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.063717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:52071 (local address 127.21.33.190:41943)
0929 10:23:21.063826 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.063830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.064069 (+ 239us) server_negotiation.cc:407] Connection header received
0929 10:23:21.064280 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.064283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.064329 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.064408 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.064828 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.065571 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.066964 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.067273 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.067377 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.068732 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.068737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.068740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.068767 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.070256 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.070763 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.070766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.070767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.070838 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.071159 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.071161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.071162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.071318 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.071390 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.071705 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.071799 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":43}
I20260929 10:23:21.072041 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.063675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41943 (local address 127.21.33.129:52071)
0929 10:23:21.063998 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.064014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.064133 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.064422 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.064425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.064632 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:21.064720 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.064725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.065734 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.065737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.066821 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.066833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.067395 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.067402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.067533 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.068068 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:21.068087 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:21.068578 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:21.070367 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.070373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.070378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.070656 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.070933 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.070938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.070941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.071050 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.071415 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:21.071418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:21.071524 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.071651 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.071808 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:21.092507 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:41943
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:21.211148 23591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:21.211460 23591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:21.211539 23591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:21.216287 23591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:21.216377 23591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:21.216452 23591 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:23:21.216495 23591 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:21.216562 23591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:21.222254 23591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:21.223392 23591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:21.224475 23591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:21.236694 23599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:21.236681 23596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:21.236681 23597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:21.237298 23591 server_base.cc:1034] running on GCE node
I20260929 10:23:21.237812 23591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:21.238561 23591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:21.239748 23591 hybrid_clock.cc:648] HybridClock initialized: now 1790677401239728 us; error 38 us; skew 500 ppm
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:21.243024 23591 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:21.244144 23591 webserver.cc:460] Webserver started at http://127.21.33.130:44667/ using document root <none> and password file <none>
I20260929 10:23:21.244781 23591 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:21.244975 23591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:21.245306 23591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:21.247483 23591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "101d23e76c3f412c9c96f760c297a1cf"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "c1b9964806daf3b091f4c3b93c319b40"
server_key_iv: "4ac257ac205fa05d2b49817ec9c95831"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.248083 23591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "101d23e76c3f412c9c96f760c297a1cf"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "c1b9964806daf3b091f4c3b93c319b40"
server_key_iv: "4ac257ac205fa05d2b49817ec9c95831"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.252245 23591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:23:21.254949 23606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:21.256327 23591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:21.256457 23591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "101d23e76c3f412c9c96f760c297a1cf"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "c1b9964806daf3b091f4c3b93c319b40"
server_key_iv: "4ac257ac205fa05d2b49817ec9c95831"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.256575 23591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:21.268299 23591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:21.271767 23591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:21.271977 23591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:21.273655 23591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:21.273715 23591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:21.273765 23591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:21.273797 23591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:21.284227 23591 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:38167
I20260929 10:23:21.284394 23721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:38167 every 8 connection(s)
I20260929 10:23:21.285126 23591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:21.290251 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 23591
I20260929 10:23:21.290378 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:21.290715 21638 external_mini_cluster.cc:1428] Setting key eb93bc622cf0d99abbdee993161bb16a
I20260929 10:23:21.290746 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.274319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:41301 (local address 127.21.33.190:41943)
0929 10:23:21.274467 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.274474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.276126 (+ 1652us) server_negotiation.cc:407] Connection header received
0929 10:23:21.277815 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.277822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.277894 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.277998 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.279817 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.280815 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.282914 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.283309 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.283419 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.287010 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.287018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.287020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.287045 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.288424 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.288930 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.288933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.288935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.289016 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.289439 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.289442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.289444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.289635 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.289745 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.290347 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.290506 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:23:21.291707 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.275374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41943 (local address 127.21.33.130:41301)
0929 10:23:21.275936 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.276002 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.277481 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.278371 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.278383 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.278893 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:21.279639 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.279661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.280992 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.280998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.282752 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.282768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.283489 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.283507 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.283749 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.284647 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:21.284674 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:21.286830 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:21.288546 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.288552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.288565 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.288823 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.289115 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.289119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.289121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.289338 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.289740 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:21.289746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:21.290020 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.290239 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.290631 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":130,"threads_started":1}
I20260929 10:23:21.293066 23722 heartbeater.cc:347] Connected to a master server at 127.21.33.190:41943
I20260929 10:23:21.293254 23722 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:21.293853 23722 heartbeater.cc:511] Master 127.21.33.190:41943 requested a full tablet report, sending...
I20260929 10:23:21.294960 23377 ts_manager.cc:194] Registered new tserver with Master: 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167)
I20260929 10:23:21.295792 23377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:41301
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:21.305357 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.295525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:44789 (local address 127.21.33.190:41943)
0929 10:23:21.295650 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.295654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.296112 (+ 458us) server_negotiation.cc:407] Connection header received
0929 10:23:21.296225 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.296231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.296278 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.296361 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.296768 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.297383 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.299019 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.299451 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.299581 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.300679 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.300685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.300692 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.300724 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.302656 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.303220 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.303225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.303230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.303330 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.304023 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.304028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.304031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.304447 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.304568 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.304979 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.305107 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:23:21.305784 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.295746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41943 (local address 127.21.33.130:44789)
0929 10:23:21.296020 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.296031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.296150 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.296376 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.296379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.296571 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:21.296660 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.296665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.297521 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.297527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.298578 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.298600 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.299615 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.299618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.299691 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.300222 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:21.300233 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:21.300525 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:21.302784 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.302788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.302789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.303054 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.303811 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.303815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.303817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.303918 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.304620 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:21.304625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:21.304772 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.304896 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.305080 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:21.323884 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:41943
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:21.437153 23731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:21.437507 23731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:21.437601 23731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:21.441659 23731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:21.441732 23731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:21.441802 23731 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 10:23:21.441843 23731 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:21.441915 23731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:21.447201 23731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:41943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:21.448282 23731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:21.449381 23731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:21.459323 23736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:21.459323 23737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:21.459468 23731 server_base.cc:1034] running on GCE node
W20260929 10:23:21.459617 23739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:21.460104 23731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:21.460690 23731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:21.461849 23731 hybrid_clock.cc:648] HybridClock initialized: now 1790677401461824 us; error 62 us; skew 500 ppm
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:21.464545 23731 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:21.465440 23731 webserver.cc:460] Webserver started at http://127.21.33.131:39233/ using document root <none> and password file <none>
I20260929 10:23:21.465967 23731 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:21.466025 23731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:21.466187 23731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:21.468168 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "787cefd336bb495ba6b07570bf485183"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "8a1d7e63a622b8841567e932207b5a9c"
server_key_iv: "27ccf83873f4b3d879b30ce84be02975"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.468675 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "787cefd336bb495ba6b07570bf485183"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "8a1d7e63a622b8841567e932207b5a9c"
server_key_iv: "27ccf83873f4b3d879b30ce84be02975"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.472288 23731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:23:21.474617 23746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:21.475889 23731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:21.476001 23731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "787cefd336bb495ba6b07570bf485183"
format_stamp: "Formatted at 2026-09-29 10:23:21 on dist-test-slave-2rbl"
server_key: "8a1d7e63a622b8841567e932207b5a9c"
server_key_iv: "27ccf83873f4b3d879b30ce84be02975"
server_key_version: "encryptionkey@0"
I20260929 10:23:21.476092 23731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:21.488240 23731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:21.491372 23731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:21.491520 23731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:21.492839 23731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:21.492962 23731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:21.493063 23731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:21.493119 23731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:21.503937 23731 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:35757
I20260929 10:23:21.504083 23861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:35757 every 8 connection(s)
I20260929 10:23:21.505052 23731 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:21.507769 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.493851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:60683 (local address 127.21.33.190:41943)
0929 10:23:21.494042 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.494046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.495023 (+ 977us) server_negotiation.cc:407] Connection header received
0929 10:23:21.496104 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.496110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.496178 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.496242 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.498102 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.498987 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.500821 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.501053 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.501185 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.504189 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.504193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.504196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.504225 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.505863 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.506496 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.506499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.506500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.506572 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.506855 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.506857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.506858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.507031 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.507119 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.507562 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.507657 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:23:21.508577 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.494400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41943 (local address 127.21.33.131:60683)
0929 10:23:21.494872 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.494913 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.495853 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.496742 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.496751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.497251 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:21.497937 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.497980 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.499136 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.499140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.500641 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.500659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.501237 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.501242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.501423 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.502056 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:21.502080 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:21.504016 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:21.506000 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.506004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.506014 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.506379 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.506653 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.506655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.506657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.506778 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.507114 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:21.507119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:21.507356 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.507512 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.507734 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260929 10:23:21.509794 23862 heartbeater.cc:347] Connected to a master server at 127.21.33.190:41943
I20260929 10:23:21.509944 23862 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:21.510416 23862 heartbeater.cc:511] Master 127.21.33.190:41943 requested a full tablet report, sending...
I20260929 10:23:21.511049 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 23731
I20260929 10:23:21.511175 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:21.511500 21638 external_mini_cluster.cc:1428] Setting key a03754498c0892ae3f4dc3180a5170b6
I20260929 10:23:21.512112 23378 ts_manager.cc:194] Registered new tserver with Master: 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:21.512849 23378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:60683
I20260929 10:23:21.513962 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:23:21.519963 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.512004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41943 (local address 127.21.33.131:53701)
0929 10:23:21.512298 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.512316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.512399 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.512624 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.512626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.512808 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:21.512932 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.512939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.513977 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.513980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.514819 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.514828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.515500 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.515503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.515573 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.516085 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:21.516103 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:21.516596 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:21.518432 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.518438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.518442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.518737 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.519011 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.519014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.519017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.519116 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.519540 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:21.519542 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:21.519625 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.519714 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.519821 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
I20260929 10:23:21.520188 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.512325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:53701 (local address 127.21.33.190:41943)
0929 10:23:21.512434 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.512438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.512452 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:21.512484 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.512486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.512522 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.512615 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.513083 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.513821 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.514992 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.515389 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.515508 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.516777 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.516780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.516782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.516802 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.518318 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.518839 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.518841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.518843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.518913 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.519216 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.519218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.519219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.519407 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.519528 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.519803 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.519934 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:21.540787 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.533078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.21.33.190:41943)
0929 10:23:21.533251 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.533255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.533275 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:23:21.533312 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.533314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.533352 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.533419 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.534149 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.534663 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.535541 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.535765 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.535844 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.538122 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.538125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.538127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.538150 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.539364 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.539726 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.539728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.539729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.539799 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.540049 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.540053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.540054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.540248 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.540361 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.540545 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.540671 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:23:21.543758 23378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:21.547480 23378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:21.562631 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.556413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:35757 (local address 127.0.0.1:41866)
0929 10:23:21.556655 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.556668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.556747 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.558316 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.558320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.558343 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:21.558444 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.558451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.559692 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.559697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.560883 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.560892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.562059 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.562063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.562199 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.562250 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.562333 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.562456 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260929 10:23:21.562777 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.556879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.21.33.129:42233)
0929 10:23:21.557163 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.557176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.557233 (+ 57us) server_negotiation.cc:407] Connection header received
0929 10:23:21.557309 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.557314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.557507 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.557957 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:21.558094 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.559270 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.561381 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.562126 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.562228 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.562368 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.562498 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.562589 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260929 10:23:21.563007 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.555666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:38167 (local address 127.0.0.1:50190)
0929 10:23:21.556036 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.556094 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.556328 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.557500 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.557507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.557534 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:21.557697 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.557706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.559300 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.559307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.561321 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.561335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.562563 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.562567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.562659 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.562687 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.562787 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.562879 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":76,"thread_start_us":103,"threads_started":1}
I20260929 10:23:21.563382 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.555634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42233 (local address 127.0.0.1:36266)
0929 10:23:21.556036 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.556094 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.556227 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.557682 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.557685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.557705 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:21.557805 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.557812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.559381 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.559384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.561083 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.561105 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.562259 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.562263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.562355 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.562379 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.562472 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.562606 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":262,"thread_start_us":104,"threads_started":1}
I20260929 10:23:21.563460 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.556474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.21.33.130:38167)
0929 10:23:21.556896 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.556908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.556936 (+ 28us) server_negotiation.cc:407] Connection header received
0929 10:23:21.557025 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.557031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.557288 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.557874 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:21.558035 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.559179 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.561466 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.562151 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.562290 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.562462 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.563201 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.563314 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260929 10:23:21.564047 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.557388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.21.33.131:35757)
0929 10:23:21.557787 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.557797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.557818 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:23:21.557890 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.557897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.558146 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.558295 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:21.558555 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.559588 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.561046 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.561917 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.562875 (+ 958us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.563031 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.563094 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.563917 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":95,"threads_started":1}
I20260929 10:23:21.566298 23796 tablet_service.cc:1467] Processing CreateTablet for tablet 202830a628b44774b63cd469ac994e29 (DEFAULT_TABLE table=test-table [id=e91dd9ae7f624de693b59184c734262f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:21.566393 23656 tablet_service.cc:1467] Processing CreateTablet for tablet 202830a628b44774b63cd469ac994e29 (DEFAULT_TABLE table=test-table [id=e91dd9ae7f624de693b59184c734262f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:21.566390 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 202830a628b44774b63cd469ac994e29 (DEFAULT_TABLE table=test-table [id=e91dd9ae7f624de693b59184c734262f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:21.567345 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202830a628b44774b63cd469ac994e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:21.567433 23656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202830a628b44774b63cd469ac994e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:21.567433 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202830a628b44774b63cd469ac994e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:21.573803 23885 tablet_bootstrap.cc:492] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Bootstrap starting.
I20260929 10:23:21.575359 23886 tablet_bootstrap.cc:492] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Bootstrap starting.
I20260929 10:23:21.576028 23885 tablet_bootstrap.cc:654] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:21.576186 23887 tablet_bootstrap.cc:492] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Bootstrap starting.
I20260929 10:23:21.576779 23885 log.cc:824] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:21.577546 23886 tablet_bootstrap.cc:654] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:21.578338 23887 tablet_bootstrap.cc:654] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:21.578850 23886 log.cc:824] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:21.579133 23885 tablet_bootstrap.cc:492] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: No bootstrap required, opened a new log
I20260929 10:23:21.579229 23887 log.cc:824] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:21.579345 23885 ts_tablet_manager.cc:1397] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:23:21.580791 23886 tablet_bootstrap.cc:492] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: No bootstrap required, opened a new log
I20260929 10:23:21.580967 23886 ts_tablet_manager.cc:1397] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:23:21.580972 23887 tablet_bootstrap.cc:492] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: No bootstrap required, opened a new log
I20260929 10:23:21.581167 23887 ts_tablet_manager.cc:1397] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:21.583259 23885 raft_consensus.cc:348] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.583515 23885 raft_consensus.cc:374] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:21.583562 23885 raft_consensus.cc:729] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787cefd336bb495ba6b07570bf485183, State: Initialized, Role: FOLLOWER
I20260929 10:23:21.584260 23885 consensus_queue.cc:260] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.584312 23887 raft_consensus.cc:348] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.584571 23887 raft_consensus.cc:374] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:21.584647 23887 raft_consensus.cc:729] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Initialized, Role: FOLLOWER
I20260929 10:23:21.584658 23886 raft_consensus.cc:348] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.584878 23886 raft_consensus.cc:374] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:21.584921 23886 raft_consensus.cc:729] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc77d2670ee44f1beac22276b9aef5d, State: Initialized, Role: FOLLOWER
I20260929 10:23:21.585121 23887 consensus_queue.cc:260] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.585264 23885 ts_tablet_manager.cc:1428] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:23:21.585328 23886 consensus_queue.cc:260] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.585588 23862 heartbeater.cc:503] Master 127.21.33.190:41943 was elected leader, sending a full tablet report...
I20260929 10:23:21.585955 23582 heartbeater.cc:503] Master 127.21.33.190:41943 was elected leader, sending a full tablet report...
I20260929 10:23:21.586206 23886 ts_tablet_manager.cc:1428] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:23:21.587168 23887 ts_tablet_manager.cc:1428] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:23:21.587507 23722 heartbeater.cc:503] Master 127.21.33.190:41943 was elected leader, sending a full tablet report...
I20260929 10:23:21.603528 23892 raft_consensus.cc:482] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:21.603814 23892 raft_consensus.cc:504] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.605113 23892 leader_election.cc:288] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:21.609505 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.605417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42233 (local address 127.21.33.130:32769)
0929 10:23:21.605543 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.605558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.605668 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.606136 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.606139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.606162 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:21.606311 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.606322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.607222 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.607225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.608299 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.608310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.609082 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.609085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.609183 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.609255 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.609331 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.609394 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 10:23:21.609581 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.605415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:32769 (local address 127.21.33.129:42233)
0929 10:23:21.605853 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.605861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.605877 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:21.605931 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.605934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.605987 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.606110 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:21.606432 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.607086 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.608431 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.608956 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.609089 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.609146 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.609368 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.609447 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260929 10:23:21.610281 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202830a628b44774b63cd469ac994e29" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d" is_pre_election: true
I20260929 10:23:21.610450 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.605755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:35757 (local address 127.21.33.130:47827)
0929 10:23:21.606151 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.606167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.606268 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.606542 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.606545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.606560 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:21.607012 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.607018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.607848 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.607852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.608836 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.608844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.609993 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.609997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.610124 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.610148 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.610234 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.610311 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":170,"threads_started":1}
I20260929 10:23:21.610458 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.605925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:47827 (local address 127.21.33.131:35757)
0929 10:23:21.606072 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.606079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.606268 (+ 189us) server_negotiation.cc:407] Connection header received
0929 10:23:21.606364 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.606367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.606438 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.606519 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:21.607149 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.607727 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.608965 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.609661 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.609822 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.609873 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.610264 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.610330 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:23:21.610648 23536 raft_consensus.cc:2454] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:21.611218 23610 leader_election.cc:302] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 6fc77d2670ee44f1beac22276b9aef5d; no voters:
I20260929 10:23:21.611286 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202830a628b44774b63cd469ac994e29" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183" is_pre_election: true
I20260929 10:23:21.611557 23816 raft_consensus.cc:2454] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:21.611555 23892 raft_consensus.cc:2781] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:21.611673 23892 raft_consensus.cc:482] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:21.611756 23892 raft_consensus.cc:3037] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:21.612943 23892 raft_consensus.cc:504] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.613348 23892 leader_election.cc:288] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Requested vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:21.613785 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202830a628b44774b63cd469ac994e29" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d"
I20260929 10:23:21.613806 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202830a628b44774b63cd469ac994e29" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183"
I20260929 10:23:21.613932 23536 raft_consensus.cc:3037] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:21.613932 23816 raft_consensus.cc:3037] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:21.615353 23816 raft_consensus.cc:2454] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 1.
I20260929 10:23:21.615353 23536 raft_consensus.cc:2454] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 1.
I20260929 10:23:21.615737 23610 leader_election.cc:302] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 787cefd336bb495ba6b07570bf485183; no voters:
I20260929 10:23:21.615950 23892 raft_consensus.cc:2781] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:21.616204 23892 raft_consensus.cc:686] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 1 LEADER]: Becoming Leader. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Running, Role: LEADER
I20260929 10:23:21.616846 23892 consensus_queue.cc:237] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.620062 23377 catalog_manager.cc:5495] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130). New cstate: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } health_report { overall_health: UNKNOWN } } }
Sep 29 10:23:21 dist-test-slave-2rbl krb5kdc[23333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677400, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:21.663667 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.655068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.21.33.190:41943)
0929 10:23:21.655220 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.655225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.655434 (+ 209us) server_negotiation.cc:407] Connection header received
0929 10:23:21.655549 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.655551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.655592 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.655672 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:21.656433 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.656954 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.657830 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.658035 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.658116 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.660538 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:21.660541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:21.660543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:21.660564 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:21.662173 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.662587 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.662589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.662591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.662662 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:21.662927 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:21.662929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:21.662930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:21.663089 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:21.663164 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.663432 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.663546 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":58}
I20260929 10:23:21.663671 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.654955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41943 (local address 127.0.0.1:46680)
0929 10:23:21.655326 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.655351 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.655461 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.655738 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.655740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.656184 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:21.656298 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.656305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.657055 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.657057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.657723 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.657732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.658165 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.658169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.658240 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.658852 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:21.658864 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:21.660405 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:21.662267 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.662276 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.662281 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.662499 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.662751 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:21.662757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:21.662760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:21.662851 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:21.663179 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:21.663184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:21.663284 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.663376 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.663494 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260929 10:23:21.667623 23377 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:46680:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:23:21.668273 23377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:21.673293 23656 tablet_service.cc:1467] Processing CreateTablet for tablet 13cc6a955f5f454c8efd1cf29dd676d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5f641ee797f4d089db2571b5430dc80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:21.673293 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 13cc6a955f5f454c8efd1cf29dd676d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5f641ee797f4d089db2571b5430dc80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:21.673668 23796 tablet_service.cc:1467] Processing CreateTablet for tablet 13cc6a955f5f454c8efd1cf29dd676d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5f641ee797f4d089db2571b5430dc80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:21.674074 23656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cc6a955f5f454c8efd1cf29dd676d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:21.674083 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cc6a955f5f454c8efd1cf29dd676d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:21.674299 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cc6a955f5f454c8efd1cf29dd676d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:21.676618 23885 tablet_bootstrap.cc:492] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: Bootstrap starting.
I20260929 10:23:21.677099 23887 tablet_bootstrap.cc:492] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: Bootstrap starting.
I20260929 10:23:21.677605 23886 tablet_bootstrap.cc:492] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: Bootstrap starting.
I20260929 10:23:21.677659 23885 tablet_bootstrap.cc:654] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:21.678184 23887 tablet_bootstrap.cc:654] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:21.678509 23886 tablet_bootstrap.cc:654] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:21.679034 23885 tablet_bootstrap.cc:492] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: No bootstrap required, opened a new log
I20260929 10:23:21.679134 23885 ts_tablet_manager.cc:1397] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 10:23:21.679492 23887 tablet_bootstrap.cc:492] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: No bootstrap required, opened a new log
I20260929 10:23:21.679591 23887 ts_tablet_manager.cc:1397] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:21.679690 23885 raft_consensus.cc:348] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.679844 23885 raft_consensus.cc:374] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:21.679890 23885 raft_consensus.cc:729] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787cefd336bb495ba6b07570bf485183, State: Initialized, Role: FOLLOWER
I20260929 10:23:21.679905 23886 tablet_bootstrap.cc:492] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: No bootstrap required, opened a new log
I20260929 10:23:21.679984 23886 ts_tablet_manager.cc:1397] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:21.680121 23887 raft_consensus.cc:348] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.680245 23887 raft_consensus.cc:374] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:21.680286 23887 raft_consensus.cc:729] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Initialized, Role: FOLLOWER
I20260929 10:23:21.680420 23887 consensus_queue.cc:260] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.680488 23886 raft_consensus.cc:348] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.680215 23885 consensus_queue.cc:260] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.680600 23886 raft_consensus.cc:374] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:21.680637 23886 raft_consensus.cc:729] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc77d2670ee44f1beac22276b9aef5d, State: Initialized, Role: FOLLOWER
I20260929 10:23:21.680737 23892 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.680764 23886 consensus_queue.cc:260] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.680858 23892 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:21.680797 23891 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.680922 23891 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:21.681015 23894 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.681154 23894 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:21.681533 23887 ts_tablet_manager.cc:1428] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:21.681874 23886 ts_tablet_manager.cc:1428] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:21.682210 23885 ts_tablet_manager.cc:1428] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
W20260929 10:23:21.756078 23863 tablet.cc:2369] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:23:21.786605 23723 tablet.cc:2369] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:23:21.809777 23583 tablet.cc:2369] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:21.870576 23892 raft_consensus.cc:482] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:21.870739 23892 raft_consensus.cc:504] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.871290 23892 leader_election.cc:288] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:21.871963 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cc6a955f5f454c8efd1cf29dd676d8" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183" is_pre_election: true
I20260929 10:23:21.871963 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cc6a955f5f454c8efd1cf29dd676d8" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d" is_pre_election: true
I20260929 10:23:21.872124 23816 raft_consensus.cc:2454] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:21.872136 23536 raft_consensus.cc:2454] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:21.872445 23610 leader_election.cc:302] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 787cefd336bb495ba6b07570bf485183; no voters:
I20260929 10:23:21.872649 23892 raft_consensus.cc:2781] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:21.872721 23892 raft_consensus.cc:482] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:21.872757 23892 raft_consensus.cc:3037] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:21.873946 23892 raft_consensus.cc:504] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.874351 23892 leader_election.cc:288] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Requested vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:21.874825 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cc6a955f5f454c8efd1cf29dd676d8" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d"
I20260929 10:23:21.874840 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cc6a955f5f454c8efd1cf29dd676d8" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183"
I20260929 10:23:21.874946 23536 raft_consensus.cc:3037] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:21.874974 23816 raft_consensus.cc:3037] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:21.875931 23816 raft_consensus.cc:2454] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 1.
I20260929 10:23:21.875932 23536 raft_consensus.cc:2454] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 1.
I20260929 10:23:21.876261 23610 leader_election.cc:302] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 6fc77d2670ee44f1beac22276b9aef5d; no voters:
I20260929 10:23:21.876444 23892 raft_consensus.cc:2781] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:21.876560 23892 raft_consensus.cc:686] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [term 1 LEADER]: Becoming Leader. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Running, Role: LEADER
I20260929 10:23:21.876690 23892 consensus_queue.cc:237] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:21.877422 23896 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: TxnStatusTablet state changed. Reason: New leader 101d23e76c3f412c9c96f760c297a1cf. Latest consensus state: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.877542 23896 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:21.877830 23908 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:23:21.878090 23908 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:23:21.878654 23378 catalog_manager.cc:5495] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130). New cstate: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:21.920789 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.916435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:38167 (local address 127.0.0.1:50194)
0929 10:23:21.916615 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.916631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.916730 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.917091 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.917093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.917100 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:21.917240 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.917250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.918277 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.918280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.918893 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.918900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.919505 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.919508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.919598 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.919650 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:21.920475 (+ 825us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:21.920491 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.920559 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.920665 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 10:23:21.920799 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.916573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.21.33.130:38167)
0929 10:23:21.916704 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.916709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.916725 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:21.916879 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.916884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.916954 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.917072 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:21.917363 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.918145 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.919013 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.919360 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.919462 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.919806 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:21.920359 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:21.920470 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.920596 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.920659 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:23:21.926805 23816 raft_consensus.cc:1261] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:21.926847 23536 raft_consensus.cc:1261] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:21.927578 23896 consensus_queue.cc:1035] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:21.927819 23892 consensus_queue.cc:1035] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:21.932519 23894 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: TxnStatusTablet state changed. Reason: New leader 101d23e76c3f412c9c96f760c297a1cf. Latest consensus state: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.932550 23891 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: TxnStatusTablet state changed. Reason: New leader 101d23e76c3f412c9c96f760c297a1cf. Latest consensus state: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.932654 23894 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:21.932680 23891 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:21.936337 23892 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.936468 23892 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:21.936990 23891 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.937099 23891 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 787cefd336bb495ba6b07570bf485183: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:21.937889 23912 mvcc.cc:205] Tried to move back new op lower bound from 7334614638284267520 to 7334614638088404992. Current Snapshot: MvccSnapshot[applied={T|T < 7334614638284267520}]
I20260929 10:23:21.938267 23892 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.938387 23892 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 101d23e76c3f412c9c96f760c297a1cf: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:21.939913 23894 tablet_replica.cc:439] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } } }
I20260929 10:23:21.940027 23894 tablet_replica.cc:442] T 13cc6a955f5f454c8efd1cf29dd676d8 P 6fc77d2670ee44f1beac22276b9aef5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:21.942291 23913 mvcc.cc:205] Tried to move back new op lower bound from 7334614638284267520 to 7334614638088404992. Current Snapshot: MvccSnapshot[applied={T|T < 7334614638284267520}]
I20260929 10:23:21.947698 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.944239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.21.33.130:38167)
0929 10:23:21.944351 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.944356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.944395 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:23:21.944567 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.944570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.944619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.944688 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:21.944940 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.945752 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.946587 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.946926 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.947030 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.947261 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:21.947346 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:21.947425 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.947516 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.947566 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 10:23:21.959393 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.955802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:37087 (local address 127.21.33.130:38167)
0929 10:23:21.955983 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:21.955987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:21.956046 (+ 59us) server_negotiation.cc:407] Connection header received
0929 10:23:21.956171 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:21.956175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:21.956220 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:21.956302 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:21.956515 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.957298 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.958148 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.958444 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.958556 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.958852 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:21.958929 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:21.958999 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:21.959170 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 10:23:21.959217 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":120}
I20260929 10:23:21.959582 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:21.955699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:38167 (local address 127.21.33.130:37087)
0929 10:23:21.955981 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:21.955993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:21.956081 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:21.956315 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:21.956317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:21.956323 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:21.956419 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.956423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.957406 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.957408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.958038 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:21.958046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:21.958554 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:21.958557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:21.958664 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:21.958707 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:21.959009 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:21.959014 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:23:21.959145 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 10:23:21.959206 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":17,"thread_start_us":101,"threads_started":1}
I20260929 10:23:21.966747 23816 raft_consensus.cc:1261] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:21.966856 23536 raft_consensus.cc:1261] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:21.967156 23896 consensus_queue.cc:1035] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:21.967305 23892 consensus_queue.cc:1035] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:21.971520 23911 mvcc.cc:205] Tried to move back new op lower bound from 7334614638452523008 to 7334614637025460224. Current Snapshot: MvccSnapshot[applied={T|T < 7334614638452523008}]
I20260929 10:23:22.005075 23378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:22.005306 23378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:22.007817 23378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Sending DeleteTablet for 3 replicas of tablet 202830a628b44774b63cd469ac994e29
I20260929 10:23:22.008554 23515 tablet_service.cc:1514] Processing DeleteTablet for tablet 202830a628b44774b63cd469ac994e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:22 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:36266
I20260929 10:23:22.008848 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet 202830a628b44774b63cd469ac994e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:22 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:41866
I20260929 10:23:22.008956 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet 202830a628b44774b63cd469ac994e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:22 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:50190
I20260929 10:23:22.009124 23927 tablet_replica.cc:330] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: stopping tablet replica
I20260929 10:23:22.009521 23929 tablet_replica.cc:330] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: stopping tablet replica
I20260929 10:23:22.009550 23927 raft_consensus.cc:2229] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:22.009780 23929 raft_consensus.cc:2229] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:22.009852 23927 raft_consensus.cc:2258] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:22.010113 23929 raft_consensus.cc:2258] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:22.010804 23378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:22.011263 23378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:22.011320 23927 ts_tablet_manager.cc:1905] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:22.011322 23929 ts_tablet_manager.cc:1905] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:22.014357 23927 ts_tablet_manager.cc:1918] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:22.014418 23929 ts_tablet_manager.cc:1918] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:22.014457 23927 log.cc:1196] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/202830a628b44774b63cd469ac994e29
I20260929 10:23:22.014523 23929 log.cc:1196] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/202830a628b44774b63cd469ac994e29
I20260929 10:23:22.014808 23927 ts_tablet_manager.cc:1939] T 202830a628b44774b63cd469ac994e29 P 6fc77d2670ee44f1beac22276b9aef5d: Deleting consensus metadata
I20260929 10:23:22.014884 23929 ts_tablet_manager.cc:1939] T 202830a628b44774b63cd469ac994e29 P 101d23e76c3f412c9c96f760c297a1cf: Deleting consensus metadata
I20260929 10:23:22.016038 23362 catalog_manager.cc:4841] TS 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167): tablet 202830a628b44774b63cd469ac994e29 (table test-table [id=e91dd9ae7f624de693b59184c734262f]) successfully deleted
I20260929 10:23:22.016026 23365 catalog_manager.cc:4841] TS 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233): tablet 202830a628b44774b63cd469ac994e29 (table test-table [id=e91dd9ae7f624de693b59184c734262f]) successfully deleted
I20260929 10:23:22.016765 23515 tablet_service.cc:1467] Processing CreateTablet for tablet e28c69359be54a9d8e77aac697acbdbc (DEFAULT_TABLE table=test-table [id=e2450e0f3c1f4b96b238b5833e0a7a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:22.017019 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28c69359be54a9d8e77aac697acbdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:22.017010 23796 tablet_service.cc:1467] Processing CreateTablet for tablet e28c69359be54a9d8e77aac697acbdbc (DEFAULT_TABLE table=test-table [id=e2450e0f3c1f4b96b238b5833e0a7a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:22.017328 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28c69359be54a9d8e77aac697acbdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:22.017726 23928 tablet_replica.cc:330] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: stopping tablet replica
I20260929 10:23:22.018029 23928 raft_consensus.cc:2229] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:22.018255 23928 raft_consensus.cc:2258] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:22.019604 23886 tablet_bootstrap.cc:492] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: Bootstrap starting.
I20260929 10:23:22.020409 23886 tablet_bootstrap.cc:654] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:22.020936 23885 tablet_bootstrap.cc:492] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: Bootstrap starting.
I20260929 10:23:22.021626 23886 tablet_bootstrap.cc:492] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: No bootstrap required, opened a new log
I20260929 10:23:22.021708 23886 ts_tablet_manager.cc:1397] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:22.021894 23885 tablet_bootstrap.cc:654] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:22.022147 23886 raft_consensus.cc:348] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.022269 23886 raft_consensus.cc:374] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:22.022315 23886 raft_consensus.cc:729] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc77d2670ee44f1beac22276b9aef5d, State: Initialized, Role: FOLLOWER
I20260929 10:23:22.022434 23886 consensus_queue.cc:260] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.022702 23886 ts_tablet_manager.cc:1428] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:22.023334 23655 tablet_service.cc:1467] Processing CreateTablet for tablet e28c69359be54a9d8e77aac697acbdbc (DEFAULT_TABLE table=test-table [id=e2450e0f3c1f4b96b238b5833e0a7a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:22.023625 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28c69359be54a9d8e77aac697acbdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:22.024587 23885 tablet_bootstrap.cc:492] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: No bootstrap required, opened a new log
I20260929 10:23:22.024679 23885 ts_tablet_manager.cc:1397] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 10:23:22.025126 23885 raft_consensus.cc:348] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.025273 23885 raft_consensus.cc:374] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:22.025309 23885 raft_consensus.cc:729] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787cefd336bb495ba6b07570bf485183, State: Initialized, Role: FOLLOWER
I20260929 10:23:22.025441 23885 consensus_queue.cc:260] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.025722 23885 ts_tablet_manager.cc:1428] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:22.027101 23928 ts_tablet_manager.cc:1905] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:22.027730 23887 tablet_bootstrap.cc:492] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: Bootstrap starting.
I20260929 10:23:22.028697 23887 tablet_bootstrap.cc:654] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:22.029999 23887 tablet_bootstrap.cc:492] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: No bootstrap required, opened a new log
I20260929 10:23:22.030081 23887 ts_tablet_manager.cc:1397] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 10:23:22.030349 23928 ts_tablet_manager.cc:1918] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:22.030427 23928 log.cc:1196] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/202830a628b44774b63cd469ac994e29
I20260929 10:23:22.030587 23887 raft_consensus.cc:348] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.030699 23887 raft_consensus.cc:374] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:22.030737 23887 raft_consensus.cc:729] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Initialized, Role: FOLLOWER
I20260929 10:23:22.030772 23928 ts_tablet_manager.cc:1939] T 202830a628b44774b63cd469ac994e29 P 787cefd336bb495ba6b07570bf485183: Deleting consensus metadata
I20260929 10:23:22.030871 23887 consensus_queue.cc:260] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.031217 23887 ts_tablet_manager.cc:1428] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:22.032624 23365 catalog_manager.cc:4841] TS 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757): tablet 202830a628b44774b63cd469ac994e29 (table test-table [id=e91dd9ae7f624de693b59184c734262f]) successfully deleted
I20260929 10:23:22.403478 23894 raft_consensus.cc:482] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:22.403709 23894 raft_consensus.cc:504] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.404824 23894 leader_election.cc:288] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:22.410359 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.405144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:38167 (local address 127.21.33.129:45941)
0929 10:23:22.405728 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:22.405743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:22.405830 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:22.406041 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:22.406043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:22.406072 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:22.406268 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.406273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.407298 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.407301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.408997 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.409009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.409932 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.409935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.410026 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.410048 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:23:22.410119 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:23:22.410178 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":495,"spinlock_wait_cycles":14848,"thread_start_us":56,"threads_started":1}
I20260929 10:23:22.410404 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.405319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45941 (local address 127.21.33.130:38167)
0929 10:23:22.405499 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:22.405504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:22.405830 (+ 326us) server_negotiation.cc:407] Connection header received
0929 10:23:22.405891 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:22.405894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:22.405949 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:22.406025 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:22.406392 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.407176 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.409131 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.409823 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.409926 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.409980 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:22.410163 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 10:23:22.410260 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 10:23:22.410604 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.405319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45137 (local address 127.21.33.131:35757)
0929 10:23:22.405602 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:22.405630 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:22.405646 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:22.405818 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:22.405823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:22.405884 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:22.405997 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:22.406393 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.407189 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.409131 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.409793 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.409923 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.409980 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:22.410383 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 10:23:22.410483 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
I20260929 10:23:22.411032 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.405144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:35757 (local address 127.21.33.129:45137)
0929 10:23:22.405517 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:22.405534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:22.405663 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:22.406039 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:22.406041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:22.406067 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:22.406265 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.406272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.407298 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.407301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.408997 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.409009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.410053 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.410056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.410185 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.410234 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 10:23:22.410336 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:23:22.410406 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":282,"spinlock_wait_cycles":384,"thread_start_us":97,"threads_started":1}
I20260929 10:23:22.411166 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28c69359be54a9d8e77aac697acbdbc" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d23e76c3f412c9c96f760c297a1cf" is_pre_election: true
I20260929 10:23:22.411427 23676 raft_consensus.cc:2454] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc77d2670ee44f1beac22276b9aef5d in term 0.
I20260929 10:23:22.411470 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28c69359be54a9d8e77aac697acbdbc" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183" is_pre_election: true
I20260929 10:23:22.411588 23816 raft_consensus.cc:2454] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc77d2670ee44f1beac22276b9aef5d in term 0.
I20260929 10:23:22.411935 23466 leader_election.cc:302] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 6fc77d2670ee44f1beac22276b9aef5d, 787cefd336bb495ba6b07570bf485183; no voters:
I20260929 10:23:22.412252 23894 raft_consensus.cc:2781] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:22.412330 23894 raft_consensus.cc:482] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:22.412386 23894 raft_consensus.cc:3037] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:22.413596 23894 raft_consensus.cc:504] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.413952 23894 leader_election.cc:288] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 election: Requested vote from peers 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:22.414317 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28c69359be54a9d8e77aac697acbdbc" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d23e76c3f412c9c96f760c297a1cf"
I20260929 10:23:22.414317 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28c69359be54a9d8e77aac697acbdbc" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183"
I20260929 10:23:22.414439 23676 raft_consensus.cc:3037] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:22.414439 23816 raft_consensus.cc:3037] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:22.415541 23816 raft_consensus.cc:2454] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc77d2670ee44f1beac22276b9aef5d in term 1.
I20260929 10:23:22.415591 23676 raft_consensus.cc:2454] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fc77d2670ee44f1beac22276b9aef5d in term 1.
I20260929 10:23:22.415853 23466 leader_election.cc:302] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 6fc77d2670ee44f1beac22276b9aef5d, 787cefd336bb495ba6b07570bf485183; no voters:
I20260929 10:23:22.416122 23894 raft_consensus.cc:2781] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:22.416332 23894 raft_consensus.cc:686] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 1 LEADER]: Becoming Leader. State: Replica: 6fc77d2670ee44f1beac22276b9aef5d, State: Running, Role: LEADER
I20260929 10:23:22.416603 23894 consensus_queue.cc:237] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:22.419121 23376 catalog_manager.cc:5495] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129). New cstate: current_term: 1 leader_uuid: "6fc77d2670ee44f1beac22276b9aef5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:22.486545 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.483089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.21.33.129:42233)
0929 10:23:22.483349 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:22.483356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:22.483388 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:23:22.483455 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:22.483457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:22.483506 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:22.483597 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:22.483806 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.484310 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.485144 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.485436 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.485518 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.485755 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:22.486227 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:22.486312 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:22.486372 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:23:22.486418 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
I20260929 10:23:22.496876 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.493872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:54057 (local address 127.21.33.130:38167)
0929 10:23:22.493992 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:22.493995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:22.494114 (+ 119us) server_negotiation.cc:407] Connection header received
0929 10:23:22.494287 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:22.494290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:22.494385 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:22.494466 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:22.494725 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.495189 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.496034 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.496222 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.496292 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.496521 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:22.496580 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:22.496629 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:22.496737 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 10:23:22.496775 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:23:22.496906 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.493762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:38167 (local address 127.21.33.129:54057)
0929 10:23:22.494037 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:22.494050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:22.494158 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:22.494495 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:22.494497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:22.494504 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:22.494598 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.494628 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.495272 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.495274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.495939 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.495948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.496310 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.496312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.496378 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.496418 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:22.496673 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:22.496677 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:23:22.496721 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 10:23:22.496776 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260929 10:23:22.502848 23816 raft_consensus.cc:1261] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Refusing update from remote peer 6fc77d2670ee44f1beac22276b9aef5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:22.503043 23676 raft_consensus.cc:1261] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Refusing update from remote peer 6fc77d2670ee44f1beac22276b9aef5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:22.503297 23936 consensus_queue.cc:1035] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:22.503486 23894 consensus_queue.cc:1035] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:22.523463 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.519782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:54285 (local address 127.21.33.129:42233)
0929 10:23:22.519886 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:22.519891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:22.519906 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:23:22.520007 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:22.520010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:22.520056 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:22.520132 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:22.520356 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.521147 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.522466 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.522678 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.522799 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.523035 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:22.523112 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:22.523191 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:22.523284 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 10:23:22.523334 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 10:23:22.523491 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:22.519574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42233 (local address 127.21.33.130:54285)
0929 10:23:22.519836 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:22.519851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:22.519937 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:22.520145 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:22.520147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:22.520153 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:22.520251 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.520256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.521359 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.521362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.522248 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:22.522257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:22.522796 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:22.522800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:22.522885 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:22.522921 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:22.523195 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:22.523203 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:23:22.523274 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:23:22.523357 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
W20260929 10:23:22.524114 23913 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 10:23:24.081255 23376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:24.081421 23376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:24.083914 23376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Sending DeleteTablet for 3 replicas of tablet e28c69359be54a9d8e77aac697acbdbc
I20260929 10:23:24.084559 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet e28c69359be54a9d8e77aac697acbdbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:24 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:41866
I20260929 10:23:24.084594 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet e28c69359be54a9d8e77aac697acbdbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:24 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:50190
I20260929 10:23:24.084594 23515 tablet_service.cc:1514] Processing DeleteTablet for tablet e28c69359be54a9d8e77aac697acbdbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:24 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:36266
I20260929 10:23:24.085070 23972 tablet_replica.cc:330] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: stopping tablet replica
I20260929 10:23:24.085295 23972 raft_consensus.cc:2229] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:24.085454 23972 raft_consensus.cc:2258] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:24.085736 23973 tablet_replica.cc:330] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: stopping tablet replica
I20260929 10:23:24.085901 23973 raft_consensus.cc:2229] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:24.086256 23973 raft_consensus.cc:2258] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:24.086393 23972 ts_tablet_manager.cc:1905] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:24.086562 23974 tablet_replica.cc:330] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: stopping tablet replica
I20260929 10:23:24.086748 23974 raft_consensus.cc:2229] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:24.086896 23974 raft_consensus.cc:2258] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:24.087702 23973 ts_tablet_manager.cc:1905] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:24.088675 23974 ts_tablet_manager.cc:1905] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:24.089545 23972 ts_tablet_manager.cc:1918] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:23:24.089629 23972 log.cc:1196] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/e28c69359be54a9d8e77aac697acbdbc
I20260929 10:23:24.089915 23972 ts_tablet_manager.cc:1939] T e28c69359be54a9d8e77aac697acbdbc P 787cefd336bb495ba6b07570bf485183: Deleting consensus metadata
I20260929 10:23:24.090806 23973 ts_tablet_manager.cc:1918] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:23:24.090811 23365 catalog_manager.cc:4841] TS 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757): tablet e28c69359be54a9d8e77aac697acbdbc (table test-table [id=e2450e0f3c1f4b96b238b5833e0a7a3a]) successfully deleted
I20260929 10:23:24.090899 23973 log.cc:1196] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/e28c69359be54a9d8e77aac697acbdbc
I20260929 10:23:24.091238 23973 ts_tablet_manager.cc:1939] T e28c69359be54a9d8e77aac697acbdbc P 6fc77d2670ee44f1beac22276b9aef5d: Deleting consensus metadata
I20260929 10:23:24.091902 23974 ts_tablet_manager.cc:1918] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:23:24.091995 23974 log.cc:1196] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/e28c69359be54a9d8e77aac697acbdbc
I20260929 10:23:24.092088 23365 catalog_manager.cc:4841] TS 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233): tablet e28c69359be54a9d8e77aac697acbdbc (table test-table [id=e2450e0f3c1f4b96b238b5833e0a7a3a]) successfully deleted
I20260929 10:23:24.092363 23974 ts_tablet_manager.cc:1939] T e28c69359be54a9d8e77aac697acbdbc P 101d23e76c3f412c9c96f760c297a1cf: Deleting consensus metadata
I20260929 10:23:24.093163 23362 catalog_manager.cc:4841] TS 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167): tablet e28c69359be54a9d8e77aac697acbdbc (table test-table [id=e2450e0f3c1f4b96b238b5833e0a7a3a]) successfully deleted
Sep 29 10:23:24 dist-test-slave-2rbl krb5kdc[23333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:23:24 dist-test-slave-2rbl krb5kdc[23333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:24.114904 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:24.106437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.21.33.190:41943)
0929 10:23:24.106766 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:24.106773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:24.106787 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:24.106864 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:24.106867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:24.106911 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:24.106989 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:24.107647 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:24.108200 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:24.108981 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:24.109307 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:24.109387 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:24.111856 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:24.111859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:24.111861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:24.111886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:24.113252 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:24.113603 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:24.113606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:24.113608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:24.113689 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:24.113955 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:24.113959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:24.113960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:24.114174 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:24.114294 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:24.114552 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 10:23:24.114667 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260929 10:23:24.117550 23376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:24.118005 23376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:24.122685 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 5e96f2ab3b2b4704beb6dea108998c92 (DEFAULT_TABLE table=test-table [id=264c5b29769d42d5bede4389e10ada31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:24.122756 23655 tablet_service.cc:1467] Processing CreateTablet for tablet 5e96f2ab3b2b4704beb6dea108998c92 (DEFAULT_TABLE table=test-table [id=264c5b29769d42d5bede4389e10ada31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:24.122893 23796 tablet_service.cc:1467] Processing CreateTablet for tablet 5e96f2ab3b2b4704beb6dea108998c92 (DEFAULT_TABLE table=test-table [id=264c5b29769d42d5bede4389e10ada31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:24.123008 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e96f2ab3b2b4704beb6dea108998c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:24.123010 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e96f2ab3b2b4704beb6dea108998c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:24.123217 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e96f2ab3b2b4704beb6dea108998c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:24.125391 23983 tablet_bootstrap.cc:492] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: Bootstrap starting.
I20260929 10:23:24.125595 23984 tablet_bootstrap.cc:492] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: Bootstrap starting.
I20260929 10:23:24.125934 23985 tablet_bootstrap.cc:492] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: Bootstrap starting.
I20260929 10:23:24.126371 23983 tablet_bootstrap.cc:654] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:24.126436 23984 tablet_bootstrap.cc:654] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:24.126835 23985 tablet_bootstrap.cc:654] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:24.127954 23984 tablet_bootstrap.cc:492] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: No bootstrap required, opened a new log
I20260929 10:23:24.127954 23985 tablet_bootstrap.cc:492] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: No bootstrap required, opened a new log
I20260929 10:23:24.127954 23983 tablet_bootstrap.cc:492] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: No bootstrap required, opened a new log
I20260929 10:23:24.128058 23985 ts_tablet_manager.cc:1397] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:24.128058 23983 ts_tablet_manager.cc:1397] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:24.128069 23984 ts_tablet_manager.cc:1397] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:24.128482 23983 raft_consensus.cc:348] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.128465 23985 raft_consensus.cc:348] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.128585 23985 raft_consensus.cc:374] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:24.128585 23983 raft_consensus.cc:374] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:24.128620 23983 raft_consensus.cc:729] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787cefd336bb495ba6b07570bf485183, State: Initialized, Role: FOLLOWER
I20260929 10:23:24.128620 23985 raft_consensus.cc:729] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Initialized, Role: FOLLOWER
I20260929 10:23:24.128557 23984 raft_consensus.cc:348] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.128683 23984 raft_consensus.cc:374] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:24.128719 23984 raft_consensus.cc:729] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc77d2670ee44f1beac22276b9aef5d, State: Initialized, Role: FOLLOWER
I20260929 10:23:24.128764 23985 consensus_queue.cc:260] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.128808 23983 consensus_queue.cc:260] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.128839 23984 consensus_queue.cc:260] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.129103 23984 ts_tablet_manager.cc:1428] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:24.129168 23983 ts_tablet_manager.cc:1428] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:24.129913 23985 ts_tablet_manager.cc:1428] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:24.405380 23967 raft_consensus.cc:482] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:24.405561 23967 raft_consensus.cc:504] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.406189 23967 leader_election.cc:288] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:24.406711 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e96f2ab3b2b4704beb6dea108998c92" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d" is_pre_election: true
I20260929 10:23:24.406852 23535 raft_consensus.cc:2454] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:24.406741 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e96f2ab3b2b4704beb6dea108998c92" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183" is_pre_election: true
I20260929 10:23:24.407044 23816 raft_consensus.cc:2454] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:24.407223 23610 leader_election.cc:302] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 6fc77d2670ee44f1beac22276b9aef5d; no voters:
I20260929 10:23:24.407406 23967 raft_consensus.cc:2781] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:24.407469 23967 raft_consensus.cc:482] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:24.407507 23967 raft_consensus.cc:3037] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:24.408644 23967 raft_consensus.cc:504] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.408970 23967 leader_election.cc:288] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Requested vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:24.409333 23535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e96f2ab3b2b4704beb6dea108998c92" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d"
I20260929 10:23:24.409368 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e96f2ab3b2b4704beb6dea108998c92" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183"
I20260929 10:23:24.409502 23816 raft_consensus.cc:3037] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:24.409529 23535 raft_consensus.cc:3037] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:24.410509 23816 raft_consensus.cc:2454] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 1.
I20260929 10:23:24.410586 23535 raft_consensus.cc:2454] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 1.
I20260929 10:23:24.410807 23610 leader_election.cc:302] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 787cefd336bb495ba6b07570bf485183; no voters:
I20260929 10:23:24.410987 23967 raft_consensus.cc:2781] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:24.411070 23967 raft_consensus.cc:686] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 1 LEADER]: Becoming Leader. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Running, Role: LEADER
I20260929 10:23:24.411257 23967 consensus_queue.cc:237] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.413161 23378 catalog_manager.cc:5495] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130). New cstate: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:24.496654 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:24.493688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.21.33.130:38167)
0929 10:23:24.493946 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:24.493953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:24.493982 (+ 29us) server_negotiation.cc:407] Connection header received
0929 10:23:24.494018 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:24.494020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:24.494060 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:24.494129 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:24.494380 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:24.494886 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:24.495710 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:24.495923 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:24.495995 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:24.496207 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:24.496283 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:24.496335 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:24.496414 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 10:23:24.496454 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":131,"threads_started":1}
I20260929 10:23:24.502429 23816 raft_consensus.cc:1261] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:24.502462 23535 raft_consensus.cc:1261] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:24.502878 23967 consensus_queue.cc:1035] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:24.503036 23953 consensus_queue.cc:1035] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:24.538091 23378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:24.538288 23378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:24.543566 23378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Sending DeleteTablet for 3 replicas of tablet 5e96f2ab3b2b4704beb6dea108998c92
I20260929 10:23:24.544169 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e96f2ab3b2b4704beb6dea108998c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:24 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:50190
I20260929 10:23:24.544195 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e96f2ab3b2b4704beb6dea108998c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:24 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:41866
I20260929 10:23:24.544363 23974 tablet_replica.cc:330] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: stopping tablet replica
I20260929 10:23:24.544339 23515 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e96f2ab3b2b4704beb6dea108998c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:24 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:36266
I20260929 10:23:24.544405 23972 tablet_replica.cc:330] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: stopping tablet replica
I20260929 10:23:24.544502 23973 tablet_replica.cc:330] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: stopping tablet replica
I20260929 10:23:24.544498 23972 raft_consensus.cc:2229] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:24.544613 23973 raft_consensus.cc:2229] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:24.544489 23974 raft_consensus.cc:2229] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:24.544727 23973 raft_consensus.cc:2258] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:24.544772 23972 raft_consensus.cc:2258] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:24.544891 23974 raft_consensus.cc:2258] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:24.545612 23974 ts_tablet_manager.cc:1905] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:24.545625 23972 ts_tablet_manager.cc:1905] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:24.545921 23378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:24.546361 23378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:24.548414 23973 ts_tablet_manager.cc:1905] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:24.548890 23974 ts_tablet_manager.cc:1918] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:24.548956 23974 log.cc:1196] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/5e96f2ab3b2b4704beb6dea108998c92
I20260929 10:23:24.549088 23972 ts_tablet_manager.cc:1918] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:24.549166 23972 log.cc:1196] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/5e96f2ab3b2b4704beb6dea108998c92
I20260929 10:23:24.549226 23974 ts_tablet_manager.cc:1939] T 5e96f2ab3b2b4704beb6dea108998c92 P 101d23e76c3f412c9c96f760c297a1cf: Deleting consensus metadata
I20260929 10:23:24.549467 23972 ts_tablet_manager.cc:1939] T 5e96f2ab3b2b4704beb6dea108998c92 P 787cefd336bb495ba6b07570bf485183: Deleting consensus metadata
I20260929 10:23:24.550036 23362 catalog_manager.cc:4841] TS 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167): tablet 5e96f2ab3b2b4704beb6dea108998c92 (table test-table [id=264c5b29769d42d5bede4389e10ada31]) successfully deleted
I20260929 10:23:24.550897 23973 ts_tablet_manager.cc:1918] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:24.550976 23973 log.cc:1196] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/5e96f2ab3b2b4704beb6dea108998c92
I20260929 10:23:24.551230 23973 ts_tablet_manager.cc:1939] T 5e96f2ab3b2b4704beb6dea108998c92 P 6fc77d2670ee44f1beac22276b9aef5d: Deleting consensus metadata
I20260929 10:23:24.551952 23365 catalog_manager.cc:4841] TS 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233): tablet 5e96f2ab3b2b4704beb6dea108998c92 (table test-table [id=264c5b29769d42d5bede4389e10ada31]) successfully deleted
I20260929 10:23:24.552101 23365 catalog_manager.cc:4841] TS 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757): tablet 5e96f2ab3b2b4704beb6dea108998c92 (table test-table [id=264c5b29769d42d5bede4389e10ada31]) successfully deleted
I20260929 10:23:24.555255 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6262cca38e4f95b8d5adcac6bf9572 (DEFAULT_TABLE table=test-table [id=bdc9b12a32414fd38560b2ebb76e36b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:24.555518 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6262cca38e4f95b8d5adcac6bf9572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:24.555591 23655 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6262cca38e4f95b8d5adcac6bf9572 (DEFAULT_TABLE table=test-table [id=bdc9b12a32414fd38560b2ebb76e36b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:24.555861 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6262cca38e4f95b8d5adcac6bf9572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:24.556017 23796 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6262cca38e4f95b8d5adcac6bf9572 (DEFAULT_TABLE table=test-table [id=bdc9b12a32414fd38560b2ebb76e36b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:24.556232 23796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6262cca38e4f95b8d5adcac6bf9572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:24.558162 23985 tablet_bootstrap.cc:492] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: Bootstrap starting.
I20260929 10:23:24.559062 23985 tablet_bootstrap.cc:654] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:24.559337 23984 tablet_bootstrap.cc:492] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d: Bootstrap starting.
I20260929 10:23:24.560325 23984 tablet_bootstrap.cc:654] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:24.560559 23985 tablet_bootstrap.cc:492] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: No bootstrap required, opened a new log
I20260929 10:23:24.560649 23985 ts_tablet_manager.cc:1397] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 10:23:24.561107 23985 raft_consensus.cc:348] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.561286 23985 raft_consensus.cc:374] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:24.561323 23985 raft_consensus.cc:729] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Initialized, Role: FOLLOWER
I20260929 10:23:24.561439 23985 consensus_queue.cc:260] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.561709 23985 ts_tablet_manager.cc:1428] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:24.562590 23983 tablet_bootstrap.cc:492] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: Bootstrap starting.
I20260929 10:23:24.563354 23984 tablet_bootstrap.cc:492] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d: No bootstrap required, opened a new log
I20260929 10:23:24.563441 23984 ts_tablet_manager.cc:1397] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 10:23:24.563496 23983 tablet_bootstrap.cc:654] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:24.563882 23984 raft_consensus.cc:348] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.563995 23984 raft_consensus.cc:374] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:24.564038 23984 raft_consensus.cc:729] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc77d2670ee44f1beac22276b9aef5d, State: Initialized, Role: FOLLOWER
I20260929 10:23:24.564163 23984 consensus_queue.cc:260] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.564453 23984 ts_tablet_manager.cc:1428] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:24.565390 23983 tablet_bootstrap.cc:492] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: No bootstrap required, opened a new log
I20260929 10:23:24.565510 23983 ts_tablet_manager.cc:1397] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:24.565886 23983 raft_consensus.cc:348] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.566009 23983 raft_consensus.cc:374] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:24.566051 23983 raft_consensus.cc:729] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787cefd336bb495ba6b07570bf485183, State: Initialized, Role: FOLLOWER
I20260929 10:23:24.566170 23983 consensus_queue.cc:260] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.566419 23983 ts_tablet_manager.cc:1428] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:24.756356 23892 raft_consensus.cc:482] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:24.756500 23892 raft_consensus.cc:504] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.756978 23892 leader_election.cc:288] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:24.757297 23965 raft_consensus.cc:482] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:24.757414 23965 raft_consensus.cc:504] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.757519 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d" is_pre_election: true
I20260929 10:23:24.757675 23536 raft_consensus.cc:2454] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:24.757858 23965 leader_election.cc:288] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:24.757874 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183" is_pre_election: true
I20260929 10:23:24.758008 23816 raft_consensus.cc:2454] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 0.
I20260929 10:23:24.757998 23610 leader_election.cc:302] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 6fc77d2670ee44f1beac22276b9aef5d; no voters:
I20260929 10:23:24.758126 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d23e76c3f412c9c96f760c297a1cf" is_pre_election: true
I20260929 10:23:24.758168 23892 raft_consensus.cc:2781] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:24.758359 23892 raft_consensus.cc:482] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:24.758423 23892 raft_consensus.cc:3037] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:24.758356 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183" is_pre_election: true
I20260929 10:23:24.758500 23816 raft_consensus.cc:2454] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fc77d2670ee44f1beac22276b9aef5d in term 0.
I20260929 10:23:24.758805 23469 leader_election.cc:302] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fc77d2670ee44f1beac22276b9aef5d, 787cefd336bb495ba6b07570bf485183; no voters:
I20260929 10:23:24.758977 23965 raft_consensus.cc:2781] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:24.759025 23965 raft_consensus.cc:482] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:24.759068 23965 raft_consensus.cc:3037] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:24.759747 23892 raft_consensus.cc:504] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.760076 23892 leader_election.cc:288] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Requested vote from peers 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:24.760278 23676 raft_consensus.cc:2379] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fc77d2670ee44f1beac22276b9aef5d in current term 1: Already voted for candidate 101d23e76c3f412c9c96f760c297a1cf in this term.
I20260929 10:23:24.760373 23965 raft_consensus.cc:504] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.760406 23536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc77d2670ee44f1beac22276b9aef5d"
I20260929 10:23:24.760607 23536 raft_consensus.cc:2379] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 101d23e76c3f412c9c96f760c297a1cf in current term 1: Already voted for candidate 6fc77d2670ee44f1beac22276b9aef5d in this term.
I20260929 10:23:24.760607 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "101d23e76c3f412c9c96f760c297a1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183"
I20260929 10:23:24.760737 23816 raft_consensus.cc:3037] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:24.760748 23965 leader_election.cc:288] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 election: Requested vote from peers 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167), 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757)
I20260929 10:23:24.761224 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "101d23e76c3f412c9c96f760c297a1cf"
I20260929 10:23:24.761348 23676 raft_consensus.cc:2379] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fc77d2670ee44f1beac22276b9aef5d in current term 1: Already voted for candidate 101d23e76c3f412c9c96f760c297a1cf in this term.
I20260929 10:23:24.761730 23815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6262cca38e4f95b8d5adcac6bf9572" candidate_uuid: "6fc77d2670ee44f1beac22276b9aef5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cefd336bb495ba6b07570bf485183"
I20260929 10:23:24.761932 23816 raft_consensus.cc:2454] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 101d23e76c3f412c9c96f760c297a1cf in term 1.
I20260929 10:23:24.762122 23469 leader_election.cc:302] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fc77d2670ee44f1beac22276b9aef5d; no voters: 101d23e76c3f412c9c96f760c297a1cf, 787cefd336bb495ba6b07570bf485183
I20260929 10:23:24.762163 23610 leader_election.cc:302] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 101d23e76c3f412c9c96f760c297a1cf, 787cefd336bb495ba6b07570bf485183; no voters: 6fc77d2670ee44f1beac22276b9aef5d
I20260929 10:23:24.762311 23892 raft_consensus.cc:2781] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:24.762377 23965 raft_consensus.cc:2726] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 10:23:24.762447 23892 raft_consensus.cc:686] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 1 LEADER]: Becoming Leader. State: Replica: 101d23e76c3f412c9c96f760c297a1cf, State: Running, Role: LEADER
I20260929 10:23:24.762596 23892 consensus_queue.cc:237] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } }
I20260929 10:23:24.764107 23377 catalog_manager.cc:5495] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130). New cstate: current_term: 1 leader_uuid: "101d23e76c3f412c9c96f760c297a1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "101d23e76c3f412c9c96f760c297a1cf" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 38167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:24.806418 23536 raft_consensus.cc:1261] T 6f6262cca38e4f95b8d5adcac6bf9572 P 6fc77d2670ee44f1beac22276b9aef5d [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:24.806427 23816 raft_consensus.cc:1261] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Refusing update from remote peer 101d23e76c3f412c9c96f760c297a1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:24.806838 23953 consensus_queue.cc:1035] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "787cefd336bb495ba6b07570bf485183" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:24.806988 23967 consensus_queue.cc:1035] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc77d2670ee44f1beac22276b9aef5d" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:23:24.821277 23951 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 10:23:26.369943 23378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:26.370090 23378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:26.372381 23378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f770622d8e846b4b7d8cd7af5268fe3: Sending DeleteTablet for 3 replicas of tablet 6f6262cca38e4f95b8d5adcac6bf9572
I20260929 10:23:26.372951 23655 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6262cca38e4f95b8d5adcac6bf9572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:26 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:50190
I20260929 10:23:26.372972 23515 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6262cca38e4f95b8d5adcac6bf9572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:26 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:36266
I20260929 10:23:26.373294 23796 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6262cca38e4f95b8d5adcac6bf9572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:26 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:41866
I20260929 10:23:26.373355 24022 tablet_replica.cc:330] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: stopping tablet replica
I20260929 10:23:26.373428 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 23450
I20260929 10:23:26.373485 24022 raft_consensus.cc:2229] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:26.374343 24022 raft_consensus.cc:2258] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:26.374608 24024 tablet_replica.cc:330] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: stopping tablet replica
I20260929 10:23:26.374758 24024 raft_consensus.cc:2229] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:26.374872 24024 raft_consensus.cc:2258] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:26.375329 24022 ts_tablet_manager.cc:1905] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:26.375499 24024 ts_tablet_manager.cc:1905] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:26.377648 24024 ts_tablet_manager.cc:1918] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:23:26.377745 24024 log.cc:1196] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/6f6262cca38e4f95b8d5adcac6bf9572
I20260929 10:23:26.377988 24024 ts_tablet_manager.cc:1939] T 6f6262cca38e4f95b8d5adcac6bf9572 P 787cefd336bb495ba6b07570bf485183: Deleting consensus metadata
I20260929 10:23:26.378825 24022 ts_tablet_manager.cc:1918] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 10:23:26.378842 23365 catalog_manager.cc:4841] TS 787cefd336bb495ba6b07570bf485183 (127.21.33.131:35757): tablet 6f6262cca38e4f95b8d5adcac6bf9572 (table test-table [id=bdc9b12a32414fd38560b2ebb76e36b3]) successfully deleted
I20260929 10:23:26.378912 24022 log.cc:1196] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/6f6262cca38e4f95b8d5adcac6bf9572
I20260929 10:23:26.379215 24022 ts_tablet_manager.cc:1939] T 6f6262cca38e4f95b8d5adcac6bf9572 P 101d23e76c3f412c9c96f760c297a1cf: Deleting consensus metadata
I20260929 10:23:26.380071 23362 catalog_manager.cc:4841] TS 101d23e76c3f412c9c96f760c297a1cf (127.21.33.130:38167): tablet 6f6262cca38e4f95b8d5adcac6bf9572 (table test-table [id=bdc9b12a32414fd38560b2ebb76e36b3]) successfully deleted
W20260929 10:23:26.382074 23365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:42233 (error 108)
I20260929 10:23:26.382855 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 23591
I20260929 10:23:26.383721 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:26.382858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42233 (local address 127.0.0.1:43292)
0929 10:23:26.383571 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:26.383636 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:42233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":98,"threads_started":1}
W20260929 10:23:26.383898 23365 catalog_manager.cc:4584] TS 6fc77d2670ee44f1beac22276b9aef5d (127.21.33.129:42233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f6262cca38e4f95b8d5adcac6bf9572: Network error: Client connection negotiation failed: client connection to 127.21.33.129:42233: connect: Connection refused (error 111)
I20260929 10:23:26.390659 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 23731
I20260929 10:23:26.396466 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 23346
2026-09-29T10:23:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6968 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:26 dist-test-slave-2rbl krb5kdc[24029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:26 dist-test-slave-2rbl krb5kdc[24029](info): set up 2 sockets
Sep 29 10:23:26 dist-test-slave-2rbl krb5kdc[24029](info): commencing operation
krb5kdc: starting...
W20260929 10:23:27.442032 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:27 dist-test-slave-2rbl krb5kdc[24029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:27.565419 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35271
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:27.675549 24042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:27.675810 24042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:27.675850 24042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:27.679852 24042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:27.679908 24042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:27.679927 24042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:27.679945 24042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:27.679960 24042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:27.685138 24042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:27.686197 24042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:27.687145 24042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:27.696215 24050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:27.696187 24047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:27.696257 24042 server_base.cc:1034] running on GCE node
W20260929 10:23:27.696200 24048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:27.697055 24042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:27.698146 24042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:27.699306 24042 hybrid_clock.cc:648] HybridClock initialized: now 1790677407699287 us; error 32 us; skew 500 ppm
Sep 29 10:23:27 dist-test-slave-2rbl krb5kdc[24029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677407, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:27.701977 24042 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:27.702778 24042 webserver.cc:460] Webserver started at http://127.21.33.190:45217/ using document root <none> and password file <none>
I20260929 10:23:27.703298 24042 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:27.703354 24042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:27.703512 24042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:27.705535 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "ad74e00b521d487593917548b678d232"
format_stamp: "Formatted at 2026-09-29 10:23:27 on dist-test-slave-2rbl"
server_key: "249b52a9dd8e74706b8154f4c76545ac"
server_key_iv: "c78d5cf0fcae18aa60cb6a49077da616"
server_key_version: "encryptionkey@0"
I20260929 10:23:27.706043 24042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "ad74e00b521d487593917548b678d232"
format_stamp: "Formatted at 2026-09-29 10:23:27 on dist-test-slave-2rbl"
server_key: "249b52a9dd8e74706b8154f4c76545ac"
server_key_iv: "c78d5cf0fcae18aa60cb6a49077da616"
server_key_version: "encryptionkey@0"
I20260929 10:23:27.709638 24042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:23:27.711951 24057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:27.713140 24042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:27.713321 24042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "ad74e00b521d487593917548b678d232"
format_stamp: "Formatted at 2026-09-29 10:23:27 on dist-test-slave-2rbl"
server_key: "249b52a9dd8e74706b8154f4c76545ac"
server_key_iv: "c78d5cf0fcae18aa60cb6a49077da616"
server_key_version: "encryptionkey@0"
I20260929 10:23:27.713438 24042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:27.722540 24042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:27.725343 24042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:27.725536 24042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:27.732754 24042 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:35271
I20260929 10:23:27.732818 24110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:35271 every 8 connection(s)
I20260929 10:23:27.733781 24042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:27.736513 24111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:27.741343 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24042
I20260929 10:23:27.741466 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:27.741763 21638 external_mini_cluster.cc:1428] Setting key 0eb17883f7a45e5a41ab7edeed4f6f86
I20260929 10:23:27.742710 24111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232: Bootstrap starting.
I20260929 10:23:27.745060 24111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:27.745849 24111 log.cc:824] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:27.747720 24111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232: No bootstrap required, opened a new log
Sep 29 10:23:27 dist-test-slave-2rbl krb5kdc[24029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:27.751289 24111 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad74e00b521d487593917548b678d232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35271 } }
I20260929 10:23:27.751536 24111 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:27.751616 24111 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad74e00b521d487593917548b678d232, State: Initialized, Role: FOLLOWER
I20260929 10:23:27.752110 24111 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad74e00b521d487593917548b678d232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35271 } }
I20260929 10:23:27.752261 24111 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:27.752351 24111 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:27.752449 24111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:27.753407 24111 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad74e00b521d487593917548b678d232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35271 } }
I20260929 10:23:27.753758 24111 leader_election.cc:302] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad74e00b521d487593917548b678d232; no voters:
I20260929 10:23:27.754108 24111 leader_election.cc:288] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:27.754181 24116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:27.754400 24116 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [term 1 LEADER]: Becoming Leader. State: Replica: ad74e00b521d487593917548b678d232, State: Running, Role: LEADER
I20260929 10:23:27.754705 24116 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad74e00b521d487593917548b678d232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35271 } }
I20260929 10:23:27.755465 24111 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:27.757000 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:27.743222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.21.33.190:35271)
0929 10:23:27.743653 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:27.743668 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:27.743704 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:23:27.744323 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:27.744338 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:27.744660 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:27.745007 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:27.745845 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:27.746870 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:27.747757 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:27.748278 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:27.748449 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:27.750935 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:27.750940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:27.750953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:27.750980 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:27.752990 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:27.753512 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:27.753520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:27.753528 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:27.753675 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:27.753986 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:27.753989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:27.753991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:27.754574 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:27.755368 (+ 794us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:27.755516 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 10:23:27.755862 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260929 10:23:27.757611 24118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad74e00b521d487593917548b678d232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad74e00b521d487593917548b678d232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35271 } } }
I20260929 10:23:27.757738 24118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:27.758164 24117 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad74e00b521d487593917548b678d232. Latest consensus state: current_term: 1 leader_uuid: "ad74e00b521d487593917548b678d232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad74e00b521d487593917548b678d232" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35271 } } }
I20260929 10:23:27.758263 24117 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232 [sys.catalog]: This master's current role is: LEADER
W20260929 10:23:27.761556 24130 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:23:27.761641 24130 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:23:27.761719 24131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:27.762706 24131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:27.768306 24131 catalog_manager.cc:1347] Generated new cluster ID: f5514ff7eb6045b689c5087328cfea1e
I20260929 10:23:27.768373 24131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:27.787127 24131 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:27.788424 24131 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:27.799763 24131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad74e00b521d487593917548b678d232: Generated new TSK 0
I20260929 10:23:27.800513 24131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:27.840663 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35271
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:27.954519 24137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:27.954804 24137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:27.954854 24137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:27.959182 24137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:27.959263 24137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:27.959327 24137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:27.964628 24137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:27.965739 24137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:27.966835 24137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:27.977035 24143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:27.977053 24142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:27.977037 24145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:27.977383 24137 server_base.cc:1034] running on GCE node
I20260929 10:23:27.977778 24137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:27.978483 24137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:27.979689 24137 hybrid_clock.cc:648] HybridClock initialized: now 1790677407979678 us; error 28 us; skew 500 ppm
Sep 29 10:23:27 dist-test-slave-2rbl krb5kdc[24029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677407, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:27.982667 24137 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:27.983474 24137 webserver.cc:460] Webserver started at http://127.21.33.129:39789/ using document root <none> and password file <none>
I20260929 10:23:27.983979 24137 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:27.984041 24137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:27.984217 24137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:27.986276 24137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "17ae9e18f74d4675affe5156d606cded"
format_stamp: "Formatted at 2026-09-29 10:23:27 on dist-test-slave-2rbl"
server_key: "18a07c0cfa6624000a4bb42f04916896"
server_key_iv: "9af3032150c8be16425210ee4dd72b7f"
server_key_version: "encryptionkey@0"
I20260929 10:23:27.986882 24137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "17ae9e18f74d4675affe5156d606cded"
format_stamp: "Formatted at 2026-09-29 10:23:27 on dist-test-slave-2rbl"
server_key: "18a07c0cfa6624000a4bb42f04916896"
server_key_iv: "9af3032150c8be16425210ee4dd72b7f"
server_key_version: "encryptionkey@0"
I20260929 10:23:27.990677 24137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:23:27.992883 24152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:27.994118 24137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:27.994252 24137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "17ae9e18f74d4675affe5156d606cded"
format_stamp: "Formatted at 2026-09-29 10:23:27 on dist-test-slave-2rbl"
server_key: "18a07c0cfa6624000a4bb42f04916896"
server_key_iv: "9af3032150c8be16425210ee4dd72b7f"
server_key_version: "encryptionkey@0"
I20260929 10:23:27.994366 24137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:28.009661 24137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:28.012849 24137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:28.013052 24137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:28.013662 24137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:28.014467 24137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:28.014513 24137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.014580 24137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:28.014611 24137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.023890 24137 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:37937
I20260929 10:23:28.023914 24265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:37937 every 8 connection(s)
I20260929 10:23:28.024770 24137 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:28.027176 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24137
I20260929 10:23:28.027297 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:28.027570 21638 external_mini_cluster.cc:1428] Setting key 328a5626d04c0e2a20619e052ebb42bc
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677407, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:23:28.038298 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:28.026554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:39671 (local address 127.21.33.190:35271)
0929 10:23:28.026694 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:28.026698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:28.027780 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 10:23:28.028756 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:28.028759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:28.028810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:28.028883 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:28.030091 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.030619 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.031751 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.031989 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.032102 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:28.034535 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:28.034546 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:28.034548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:28.034570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:28.036062 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:28.036635 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:28.036639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:28.036641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:28.036723 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:28.037103 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:28.037106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:28.037108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:28.037367 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:28.037475 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:28.037991 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 10:23:28.038119 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:23:28.039302 24268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:28.026910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35271 (local address 127.21.33.129:39671)
0929 10:23:28.027607 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:28.027660 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:28.028552 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:28.029048 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:28.029055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:28.029467 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:28.029945 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:28.029958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.030764 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.030767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:28.031623 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:28.031632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.032138 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.032143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:28.032313 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:28.032902 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:28.032917 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:28.034372 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:28.036204 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:28.036210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:28.036224 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:28.036529 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:28.036824 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:28.036827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:28.036829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:28.036966 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:28.037474 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:28.037481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:28.037758 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 10:23:28.037951 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 10:23:28.038362 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":136,"threads_started":1}
I20260929 10:23:28.040808 24266 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35271
I20260929 10:23:28.041045 24266 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:28.041658 24266 heartbeater.cc:511] Master 127.21.33.190:35271 requested a full tablet report, sending...
I20260929 10:23:28.043259 24074 ts_manager.cc:194] Registered new tserver with Master: 17ae9e18f74d4675affe5156d606cded (127.21.33.129:37937)
I20260929 10:23:28.044379 24074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:39671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:28.054464 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35271
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:28.164305 24271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:28.164568 24271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:28.164610 24271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:28.168785 24271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:28.168854 24271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:28.168913 24271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:28.174552 24271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:28.175603 24271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:28.176636 24271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:28.186328 24279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:28.186326 24277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:28.186340 24276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:28.186546 24271 server_base.cc:1034] running on GCE node
I20260929 10:23:28.187034 24271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:28.187628 24271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:28.188784 24271 hybrid_clock.cc:648] HybridClock initialized: now 1790677408188749 us; error 51 us; skew 500 ppm
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:28.191497 24271 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:28.192346 24271 webserver.cc:460] Webserver started at http://127.21.33.130:38519/ using document root <none> and password file <none>
I20260929 10:23:28.192905 24271 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:28.192965 24271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:28.193243 24271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:28.195292 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "17f23b77063542e6b6c5d3cd9b776803"
format_stamp: "Formatted at 2026-09-29 10:23:28 on dist-test-slave-2rbl"
server_key: "0090bcd94ffecde9d0a45cbbb3d67798"
server_key_iv: "aab9d3e9c99be9af060b287552cdd4e2"
server_key_version: "encryptionkey@0"
I20260929 10:23:28.195863 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "17f23b77063542e6b6c5d3cd9b776803"
format_stamp: "Formatted at 2026-09-29 10:23:28 on dist-test-slave-2rbl"
server_key: "0090bcd94ffecde9d0a45cbbb3d67798"
server_key_iv: "aab9d3e9c99be9af060b287552cdd4e2"
server_key_version: "encryptionkey@0"
I20260929 10:23:28.199683 24271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:23:28.202170 24286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.203399 24271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:28.203526 24271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "17f23b77063542e6b6c5d3cd9b776803"
format_stamp: "Formatted at 2026-09-29 10:23:28 on dist-test-slave-2rbl"
server_key: "0090bcd94ffecde9d0a45cbbb3d67798"
server_key_iv: "aab9d3e9c99be9af060b287552cdd4e2"
server_key_version: "encryptionkey@0"
I20260929 10:23:28.203631 24271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:28.222119 24271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:28.225067 24271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:28.225299 24271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:28.225919 24271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:28.226770 24271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:28.226826 24271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.226899 24271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:28.226938 24271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.236047 24271 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:42131
I20260929 10:23:28.236070 24400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:42131 every 8 connection(s)
I20260929 10:23:28.236963 24271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:28.240970 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24271
I20260929 10:23:28.241115 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:28.241505 21638 external_mini_cluster.cc:1428] Setting key 2aba96f365d4e7c3fa8e769199fc5db2
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:28.252202 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:28.238886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:60231 (local address 127.21.33.190:35271)
0929 10:23:28.239034 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:28.239037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:28.240107 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 10:23:28.241259 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:28.241262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:28.241317 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:28.241394 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:28.242898 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.243661 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.245143 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.245431 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.245516 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:28.248240 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:28.248246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:28.248252 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:28.248286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:28.250057 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:28.250576 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:28.250582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:28.250585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:28.250682 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:28.251042 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:28.251049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:28.251053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:28.251312 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:28.251401 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:28.251964 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 10:23:28.252074 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:23:28.253090 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:28.239283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35271 (local address 127.21.33.130:60231)
0929 10:23:28.239956 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:28.239999 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:28.240911 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:28.241616 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:28.241626 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:28.242085 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:28.242747 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:28.242770 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.243810 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.243813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:28.245007 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:28.245022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.245597 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.245602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:28.245820 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:28.246498 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:28.246514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:28.248051 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:28.250181 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:28.250186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:28.250199 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:28.250464 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:28.250790 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:28.250793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:28.250802 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:28.250940 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:28.251416 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:28.251421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:28.251720 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 10:23:28.251898 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 10:23:28.252165 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":112,"threads_started":1}
I20260929 10:23:28.254336 24401 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:23:28.254643 24401 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:28.255280 24401 heartbeater.cc:511] Master 127.21.33.190:35271 requested a full tablet report, sending...
I20260929 10:23:28.256558 24074 ts_manager.cc:194] Registered new tserver with Master: 17f23b77063542e6b6c5d3cd9b776803 (127.21.33.130:42131)
I20260929 10:23:28.257440 24074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:60231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:28.272729 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35271
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:28.383955 24406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:28.384215 24406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:28.384296 24406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:28.388420 24406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:28.388489 24406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:28.388598 24406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:28.393992 24406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:28.395046 24406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:28.396075 24406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:28.405741 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:28.405841 24406 server_base.cc:1034] running on GCE node
W20260929 10:23:28.405730 24411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:28.405730 24414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:28.406540 24406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:28.407196 24406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:28.408390 24406 hybrid_clock.cc:648] HybridClock initialized: now 1790677408408347 us; error 59 us; skew 500 ppm
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:28.411114 24406 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:28.412060 24406 webserver.cc:460] Webserver started at http://127.21.33.131:38811/ using document root <none> and password file <none>
I20260929 10:23:28.412578 24406 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:28.412637 24406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:28.412807 24406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:28.414875 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "9e4c7cb116ad4677a72c3bdebc1dcf1a"
format_stamp: "Formatted at 2026-09-29 10:23:28 on dist-test-slave-2rbl"
server_key: "df581ac3d604fecd5fd2be714257cbf2"
server_key_iv: "36495c66d8ea6a1a8c4b062cf3adb9e5"
server_key_version: "encryptionkey@0"
I20260929 10:23:28.415383 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "9e4c7cb116ad4677a72c3bdebc1dcf1a"
format_stamp: "Formatted at 2026-09-29 10:23:28 on dist-test-slave-2rbl"
server_key: "df581ac3d604fecd5fd2be714257cbf2"
server_key_iv: "36495c66d8ea6a1a8c4b062cf3adb9e5"
server_key_version: "encryptionkey@0"
I20260929 10:23:28.419094 24406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 10:23:28.421379 24421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.422552 24406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 10:23:28.422678 24406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "9e4c7cb116ad4677a72c3bdebc1dcf1a"
format_stamp: "Formatted at 2026-09-29 10:23:28 on dist-test-slave-2rbl"
server_key: "df581ac3d604fecd5fd2be714257cbf2"
server_key_iv: "36495c66d8ea6a1a8c4b062cf3adb9e5"
server_key_version: "encryptionkey@0"
I20260929 10:23:28.422793 24406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:28.449549 24406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:28.452389 24406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:28.452594 24406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:28.453186 24406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:28.454005 24406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:28.454048 24406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.454116 24406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:28.454154 24406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:28.463285 24406 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:35791
I20260929 10:23:28.463315 24534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:35791 every 8 connection(s)
I20260929 10:23:28.464201 24406 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:28.469370 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24406
I20260929 10:23:28.469511 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:28.469812 21638 external_mini_cluster.cc:1428] Setting key f57230e9fc2ed4e775f8945b687de1d8
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677408, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:28.477155 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:28.465952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:53537 (local address 127.21.33.190:35271)
0929 10:23:28.466089 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:28.466093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:28.467142 (+ 1049us) server_negotiation.cc:407] Connection header received
0929 10:23:28.467888 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:28.467890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:28.467937 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:28.468015 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:28.469157 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.469904 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.471326 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.471556 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.471668 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:28.474165 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:28.474168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:28.474170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:28.474194 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:28.475463 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:28.475912 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:28.475914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:28.475916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:28.475988 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:28.476275 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:28.476277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:28.476278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:28.476440 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:28.476515 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:28.476942 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 10:23:28.477038 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:23:28.478026 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:28.466294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35271 (local address 127.21.33.131:53537)
0929 10:23:28.466922 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:28.466962 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:28.467705 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:28.468174 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:28.468180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:28.468524 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:28.468986 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:28.468999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.470039 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.470044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:28.471200 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:28.471210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:28.471702 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:28.471706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:28.471916 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:28.472538 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:28.472554 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:28.474014 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:28.475567 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:28.475571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:28.475582 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:28.475828 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:28.476066 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:28.476068 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:28.476069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:28.476199 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:28.476526 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:28.476531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:28.476749 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 10:23:28.476901 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 10:23:28.477141 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":110,"threads_started":1}
I20260929 10:23:28.479113 24535 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35271
I20260929 10:23:28.479357 24535 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:28.479836 24535 heartbeater.cc:511] Master 127.21.33.190:35271 requested a full tablet report, sending...
I20260929 10:23:28.480820 24074 ts_manager.cc:194] Registered new tserver with Master: 9e4c7cb116ad4677a72c3bdebc1dcf1a (127.21.33.131:35791)
I20260929 10:23:28.481429 24074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:53537
I20260929 10:23:28.483556 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:23:28.495957 24114 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:28.494621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.21.33.190:35271)
0929 10:23:28.494771 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:28.494788 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:28.495001 (+ 213us) server_negotiation.cc:407] Connection header received
0929 10:23:28.495102 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:28.495105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:28.495144 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:28.495216 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:28.495853 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39036: BlockingRecv error: recv got EOF from 127.0.0.1:39036 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:23:28.497057 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24137
I20260929 10:23:28.502516 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24271
I20260929 10:23:28.507719 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24406
I20260929 10:23:28.512904 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24042
2026-09-29T10:23:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2115 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24547](info): set up 2 sockets
Sep 29 10:23:28 dist-test-slave-2rbl krb5kdc[24547](info): commencing operation
krb5kdc: starting...
W20260929 10:23:29.549705 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:29 dist-test-slave-2rbl krb5kdc[24547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:29.674168 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33925
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:33925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:29.783664 24560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:29.783946 24560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:29.783986 24560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:29.788033 24560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:29.788095 24560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:29.788115 24560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:29.788130 24560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:29.788146 24560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:29.793481 24560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:33925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:29.794519 24560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:29.795557 24560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:29.804271 24566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:29.804265 24565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:29.804265 24568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:29.804330 24560 server_base.cc:1034] running on GCE node
I20260929 10:23:29.805106 24560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:29.806216 24560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:29.807384 24560 hybrid_clock.cc:648] HybridClock initialized: now 1790677409807351 us; error 46 us; skew 500 ppm
Sep 29 10:23:29 dist-test-slave-2rbl krb5kdc[24547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677409, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:29.810204 24560 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:29.811021 24560 webserver.cc:460] Webserver started at http://127.21.33.190:33057/ using document root <none> and password file <none>
I20260929 10:23:29.811542 24560 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:29.811599 24560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:29.811764 24560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:29.813838 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "548a57b9b3a44b6a8b734e8b114df19a"
format_stamp: "Formatted at 2026-09-29 10:23:29 on dist-test-slave-2rbl"
server_key: "04655130bf2d00e4dc3c18ee5a4d9a37"
server_key_iv: "2f6624c29a0c715934566eace655369e"
server_key_version: "encryptionkey@0"
I20260929 10:23:29.814364 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "548a57b9b3a44b6a8b734e8b114df19a"
format_stamp: "Formatted at 2026-09-29 10:23:29 on dist-test-slave-2rbl"
server_key: "04655130bf2d00e4dc3c18ee5a4d9a37"
server_key_iv: "2f6624c29a0c715934566eace655369e"
server_key_version: "encryptionkey@0"
I20260929 10:23:29.817953 24560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:23:29.820314 24575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:29.821631 24560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:29.821762 24560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "548a57b9b3a44b6a8b734e8b114df19a"
format_stamp: "Formatted at 2026-09-29 10:23:29 on dist-test-slave-2rbl"
server_key: "04655130bf2d00e4dc3c18ee5a4d9a37"
server_key_iv: "2f6624c29a0c715934566eace655369e"
server_key_version: "encryptionkey@0"
I20260929 10:23:29.821862 24560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:29.836534 24560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:29.839560 24560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:29.839775 24560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:29.846817 24560 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:33925
I20260929 10:23:29.847086 24627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:33925 every 8 connection(s)
I20260929 10:23:29.847832 24560 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:29.850163 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24560
I20260929 10:23:29.850304 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:29.850607 21638 external_mini_cluster.cc:1428] Setting key 2e4f7b1a95072acef61632c47067b01d
I20260929 10:23:29.850888 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:29.858031 24628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a: Bootstrap starting.
Sep 29 10:23:29 dist-test-slave-2rbl krb5kdc[24547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:29.860169 24628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:29.861021 24628 log.cc:824] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:29.863430 24628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a: No bootstrap required, opened a new log
I20260929 10:23:29.866158 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:29.852006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.21.33.190:33925)
0929 10:23:29.852454 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:29.852473 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:29.852512 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:23:29.853328 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:29.853351 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:29.853769 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:29.854134 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:29.854886 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:29.856357 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:29.857274 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:29.857936 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:29.858102 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:29.860756 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:29.860762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:29.860777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:29.860814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:29.862894 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:29.863359 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:29.863364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:29.863370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:29.863510 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:29.863840 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:29.863843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:29.863845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:29.864341 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:29.864527 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:29.864807 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 10:23:29.865097 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260929 10:23:29.867923 24628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a57b9b3a44b6a8b734e8b114df19a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33925 } }
I20260929 10:23:29.868197 24628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:29.868259 24628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548a57b9b3a44b6a8b734e8b114df19a, State: Initialized, Role: FOLLOWER
I20260929 10:23:29.868719 24628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a57b9b3a44b6a8b734e8b114df19a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33925 } }
I20260929 10:23:29.868882 24628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:29.868963 24628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:29.869060 24628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:29.869385 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:23:29.870129 24628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a57b9b3a44b6a8b734e8b114df19a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33925 } }
I20260929 10:23:29.870486 24628 leader_election.cc:302] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 548a57b9b3a44b6a8b734e8b114df19a; no voters:
I20260929 10:23:29.870841 24628 leader_election.cc:288] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:29.870918 24633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:29.871197 24633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [term 1 LEADER]: Becoming Leader. State: Replica: 548a57b9b3a44b6a8b734e8b114df19a, State: Running, Role: LEADER
I20260929 10:23:29.871544 24633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a57b9b3a44b6a8b734e8b114df19a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33925 } }
I20260929 10:23:29.872138 24628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:29.873148 24636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "548a57b9b3a44b6a8b734e8b114df19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a57b9b3a44b6a8b734e8b114df19a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33925 } } }
I20260929 10:23:29.873351 24636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:29.873629 24636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 548a57b9b3a44b6a8b734e8b114df19a. Latest consensus state: current_term: 1 leader_uuid: "548a57b9b3a44b6a8b734e8b114df19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548a57b9b3a44b6a8b734e8b114df19a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33925 } } }
I20260929 10:23:29.873790 24636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:29.874750 24643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:29.877790 24643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:29.883054 24643 catalog_manager.cc:1347] Generated new cluster ID: 7d06fe9b27cb410badbd37c249cd947e
I20260929 10:23:29.883154 24643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:29.894552 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:29.888915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.21.33.190:33925)
0929 10:23:29.889032 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:29.889036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:29.889354 (+ 318us) server_negotiation.cc:407] Connection header received
0929 10:23:29.889487 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:29.889490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:29.889541 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:29.889621 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:29.890228 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:29.890997 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:29.892177 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:29.892512 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:29.892604 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:29.893558 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:29.893562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:29.893565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:29.894106 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:29.894139 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:29.894157 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:29.894238 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:29.894357 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 10:23:29.894412 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:23:29.903244 24643 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:29.904134 24643 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:29.910841 24643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 548a57b9b3a44b6a8b734e8b114df19a: Generated new TSK 0
I20260929 10:23:29.911726 24643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:23:29.917675 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24560
2026-09-29T10:23:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1401 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:29 dist-test-slave-2rbl krb5kdc[24661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:29 dist-test-slave-2rbl krb5kdc[24661](info): set up 2 sockets
Sep 29 10:23:29 dist-test-slave-2rbl krb5kdc[24661](info): commencing operation
krb5kdc: starting...
W20260929 10:23:30.957716 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:31.084771 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35031
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:31.202912 24674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:31.203258 24674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:31.203338 24674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:31.207522 24674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:31.207631 24674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:31.207675 24674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:31.207742 24674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:31.207787 24674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:31.213403 24674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:31.214548 24674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:31.215677 24674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:31.224699 24680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:31.224702 24682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:31.224646 24679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:31.224889 24674 server_base.cc:1034] running on GCE node
I20260929 10:23:31.225656 24674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:31.226729 24674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:31.227926 24674 hybrid_clock.cc:648] HybridClock initialized: now 1790677411227890 us; error 48 us; skew 500 ppm
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:31.230810 24674 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:31.231673 24674 webserver.cc:460] Webserver started at http://127.21.33.190:34083/ using document root <none> and password file <none>
I20260929 10:23:31.232204 24674 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:31.232271 24674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:31.232436 24674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:31.234520 24674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "2a14c29db54e43abadc50625a506612e"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "85641dcec30d8436d4a0acab7add3cdc"
server_key_iv: "6d278cd1d5674db309512d62b8cfb8ce"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.235040 24674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "2a14c29db54e43abadc50625a506612e"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "85641dcec30d8436d4a0acab7add3cdc"
server_key_iv: "6d278cd1d5674db309512d62b8cfb8ce"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.238813 24674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:23:31.241292 24689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.242555 24674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:31.242660 24674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "2a14c29db54e43abadc50625a506612e"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "85641dcec30d8436d4a0acab7add3cdc"
server_key_iv: "6d278cd1d5674db309512d62b8cfb8ce"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.242750 24674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:31.257616 24674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:31.260948 24674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:31.261137 24674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:31.268170 24674 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:35031
I20260929 10:23:31.268469 24741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:35031 every 8 connection(s)
I20260929 10:23:31.269168 24674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:31.271190 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24674
I20260929 10:23:31.271322 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:31.271605 21638 external_mini_cluster.cc:1428] Setting key af4e37e4e927ae1cfe8a868150f716f6
I20260929 10:23:31.272518 24742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:31.278533 24742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e: Bootstrap starting.
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:31.280968 24742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:31.281791 24742 log.cc:824] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:31.283707 24742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e: No bootstrap required, opened a new log
I20260929 10:23:31.285465 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:31.273056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.21.33.190:35031)
0929 10:23:31.273548 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:31.273563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:31.273599 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:23:31.274247 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:31.274264 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:31.274594 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:31.274941 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:31.275733 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.276716 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.277633 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.278103 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.278283 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:31.280783 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:31.280788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:31.280803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:31.280831 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:31.282620 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.283073 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.283078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.283082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.283191 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.283473 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.283476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.283478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.283913 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:31.284116 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:31.284413 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 10:23:31.284696 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":188,"threads_started":1}
I20260929 10:23:31.287974 24742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a14c29db54e43abadc50625a506612e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35031 } }
I20260929 10:23:31.288218 24742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:31.288297 24742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a14c29db54e43abadc50625a506612e, State: Initialized, Role: FOLLOWER
I20260929 10:23:31.288735 24742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a14c29db54e43abadc50625a506612e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35031 } }
I20260929 10:23:31.288893 24742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:31.288982 24742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:31.289104 24742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:31.290043 24742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a14c29db54e43abadc50625a506612e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35031 } }
I20260929 10:23:31.290390 24742 leader_election.cc:302] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a14c29db54e43abadc50625a506612e; no voters:
I20260929 10:23:31.290786 24742 leader_election.cc:288] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:31.290874 24747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:31.291146 24747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [term 1 LEADER]: Becoming Leader. State: Replica: 2a14c29db54e43abadc50625a506612e, State: Running, Role: LEADER
I20260929 10:23:31.291461 24747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a14c29db54e43abadc50625a506612e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35031 } }
I20260929 10:23:31.292042 24742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:31.293396 24748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a14c29db54e43abadc50625a506612e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a14c29db54e43abadc50625a506612e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35031 } } }
I20260929 10:23:31.293551 24748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:31.293772 24749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a14c29db54e43abadc50625a506612e. Latest consensus state: current_term: 1 leader_uuid: "2a14c29db54e43abadc50625a506612e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a14c29db54e43abadc50625a506612e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35031 } } }
I20260929 10:23:31.293869 24749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:31.294183 24756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:31.297034 24756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:31.302547 24756 catalog_manager.cc:1347] Generated new cluster ID: 7c10cc24a9da49f9b3a0c03efc61c611
I20260929 10:23:31.302619 24756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:31.311497 24756 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:31.312361 24756 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:31.319015 24756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a14c29db54e43abadc50625a506612e: Generated new TSK 0
I20260929 10:23:31.319645 24756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:31.352445 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35031
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:31.466274 24768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:31.466542 24768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:31.466590 24768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:31.470811 24768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:31.470875 24768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:31.470974 24768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:31.476229 24768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:31.477347 24768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:31.478442 24768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:31.488039 24774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:31.488032 24773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:31.488272 24768 server_base.cc:1034] running on GCE node
W20260929 10:23:31.488095 24776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:31.488747 24768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:31.489416 24768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:31.490619 24768 hybrid_clock.cc:648] HybridClock initialized: now 1790677411490602 us; error 27 us; skew 500 ppm
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:31.498342 24768 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:31.499199 24768 webserver.cc:460] Webserver started at http://127.21.33.129:41975/ using document root <none> and password file <none>
I20260929 10:23:31.499763 24768 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:31.499825 24768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:31.500084 24768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:31.502146 24768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "3a5fce792ccb48ad84715925b4821843"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "8001fd7f09af66def4beed92044653a8"
server_key_iv: "176825ebf365e70cbeba379623c8f9c3"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.502696 24768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "3a5fce792ccb48ad84715925b4821843"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "8001fd7f09af66def4beed92044653a8"
server_key_iv: "176825ebf365e70cbeba379623c8f9c3"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.506304 24768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 10:23:31.508526 24783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.509821 24768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:31.509948 24768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "3a5fce792ccb48ad84715925b4821843"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "8001fd7f09af66def4beed92044653a8"
server_key_iv: "176825ebf365e70cbeba379623c8f9c3"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.510047 24768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:31.525717 24768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:31.529012 24768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:31.529271 24768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:31.529878 24768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:31.530692 24768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:31.530738 24768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.530802 24768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:31.530834 24768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.539974 24768 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:43877
I20260929 10:23:31.539999 24896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:43877 every 8 connection(s)
I20260929 10:23:31.540854 24768 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:31.548981 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24768
I20260929 10:23:31.549154 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:31.549551 21638 external_mini_cluster.cc:1428] Setting key aa2bd75523854cf4de94c7b82e6c7982
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:31.554781 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:31.542689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45467 (local address 127.21.33.190:35031)
0929 10:23:31.542829 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:31.542833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:31.543600 (+ 767us) server_negotiation.cc:407] Connection header received
0929 10:23:31.544870 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:31.544873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:31.544918 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:31.544991 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:31.546228 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.546773 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.547968 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.548302 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.548382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:31.551130 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:31.551139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:31.551141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:31.551163 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:31.552637 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.553153 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.553158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.553161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.553278 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.553599 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.553603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.553604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.553855 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:31.553935 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:31.554403 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 10:23:31.554550 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:23:31.555506 24899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:31.543024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35031 (local address 127.21.33.129:45467)
0929 10:23:31.543453 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:31.543494 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:31.544663 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:31.545156 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:31.545162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:31.545575 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:31.546103 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:31.546116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.546888 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.546890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:31.547857 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:31.547866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.548431 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.548433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:31.548573 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:31.549235 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:31.549257 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:31.550960 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:31.552785 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:31.552789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:31.552800 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:31.553053 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:31.553368 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:31.553371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:31.553372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:31.553502 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:31.553944 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:31.553949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:31.554168 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 10:23:31.554324 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 10:23:31.554569 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260929 10:23:31.556804 24897 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35031
I20260929 10:23:31.557050 24897 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:31.557582 24897 heartbeater.cc:511] Master 127.21.33.190:35031 requested a full tablet report, sending...
I20260929 10:23:31.559101 24706 ts_manager.cc:194] Registered new tserver with Master: 3a5fce792ccb48ad84715925b4821843 (127.21.33.129:43877)
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:23:31.560246 24706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:45467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:31.579596 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35031
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:31.688906 24902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:31.689184 24902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:31.689255 24902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:31.693528 24902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:31.693584 24902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:31.693681 24902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:31.698946 24902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:31.700047 24902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:31.701153 24902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:31.710774 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:31.710837 24907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:31.710897 24902 server_base.cc:1034] running on GCE node
W20260929 10:23:31.710860 24908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:31.711427 24902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:31.712054 24902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:31.713287 24902 hybrid_clock.cc:648] HybridClock initialized: now 1790677411713270 us; error 28 us; skew 500 ppm
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:31.716059 24902 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:31.716961 24902 webserver.cc:460] Webserver started at http://127.21.33.130:38485/ using document root <none> and password file <none>
I20260929 10:23:31.717561 24902 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:31.717621 24902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:31.717780 24902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:31.719784 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "e415ad54ec184478b91400090e8db2d7"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "ca28728f4239f5cbbf2b0fbc42c9c3da"
server_key_iv: "beb94e8bfd6936127502f63e50999661"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.720287 24902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "e415ad54ec184478b91400090e8db2d7"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "ca28728f4239f5cbbf2b0fbc42c9c3da"
server_key_iv: "beb94e8bfd6936127502f63e50999661"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.723783 24902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 10:23:31.726069 24917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.727249 24902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:31.727378 24902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "e415ad54ec184478b91400090e8db2d7"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "ca28728f4239f5cbbf2b0fbc42c9c3da"
server_key_iv: "beb94e8bfd6936127502f63e50999661"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.727479 24902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:31.738092 24902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:31.740810 24902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:31.741003 24902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:31.741626 24902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:31.742408 24902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:31.742452 24902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.742517 24902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:31.742549 24902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.751662 24902 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:38847
I20260929 10:23:31.751698 25030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:38847 every 8 connection(s)
I20260929 10:23:31.752542 24902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:31.756072 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 24902
I20260929 10:23:31.756219 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:31.756516 21638 external_mini_cluster.cc:1428] Setting key e00258a56813dfe19501259668e3e9f0
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:31.765954 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:31.754292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:35815 (local address 127.21.33.190:35031)
0929 10:23:31.754432 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:31.754437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:31.755183 (+ 746us) server_negotiation.cc:407] Connection header received
0929 10:23:31.756617 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:31.756621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:31.756710 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:31.756813 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:31.758010 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.758540 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.759663 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.759891 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.760002 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:31.762433 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:31.762437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:31.762440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:31.762471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:31.763961 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.764455 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.764459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.764461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.764541 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.764875 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.764879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.764881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.765050 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:31.765166 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:31.765673 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 10:23:31.765791 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:23:31.766763 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:31.754618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35031 (local address 127.21.33.130:35815)
0929 10:23:31.755036 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:31.755077 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:31.756408 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:31.756983 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:31.756990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:31.757412 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:31.757870 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:31.757882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.758662 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.758667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:31.759530 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:31.759539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.760033 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.760037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:31.760217 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:31.760812 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:31.760827 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:31.762278 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:31.764087 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:31.764092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:31.764104 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:31.764353 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:31.764632 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:31.764644 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:31.764646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:31.764782 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:31.765154 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:31.765160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:31.765437 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 10:23:31.765619 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 10:23:31.765930 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:23:31.767870 25031 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35031
I20260929 10:23:31.768112 25031 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:31.768597 25031 heartbeater.cc:511] Master 127.21.33.190:35031 requested a full tablet report, sending...
I20260929 10:23:31.769717 24706 ts_manager.cc:194] Registered new tserver with Master: e415ad54ec184478b91400090e8db2d7 (127.21.33.130:38847)
I20260929 10:23:31.770282 24706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:35815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:31.784206 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35031
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:31.893771 25036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:31.894026 25036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:31.894068 25036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:31.898098 25036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:31.898161 25036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:31.898223 25036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:31.903431 25036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:31.904398 25036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:31.905364 25036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:31.914940 25044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:31.914976 25042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:31.915005 25036 server_base.cc:1034] running on GCE node
W20260929 10:23:31.914948 25041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:31.915516 25036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:31.916145 25036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:31.917294 25036 hybrid_clock.cc:648] HybridClock initialized: now 1790677411917261 us; error 44 us; skew 500 ppm
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:31.920030 25036 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:31.920842 25036 webserver.cc:460] Webserver started at http://127.21.33.131:37227/ using document root <none> and password file <none>
I20260929 10:23:31.921442 25036 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:31.921501 25036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:31.921723 25036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:31.923693 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "74d3d71711b542bbbc93b284ae3c3f0b"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "bf1217359b4b9577ac319cd2f08e2421"
server_key_iv: "a9204285b3a193755d9606a06ac70aa7"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.924249 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "74d3d71711b542bbbc93b284ae3c3f0b"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "bf1217359b4b9577ac319cd2f08e2421"
server_key_iv: "a9204285b3a193755d9606a06ac70aa7"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.927884 25036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:23:31.930308 25051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.931790 25036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:23:31.931911 25036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "74d3d71711b542bbbc93b284ae3c3f0b"
format_stamp: "Formatted at 2026-09-29 10:23:31 on dist-test-slave-2rbl"
server_key: "bf1217359b4b9577ac319cd2f08e2421"
server_key_iv: "a9204285b3a193755d9606a06ac70aa7"
server_key_version: "encryptionkey@0"
I20260929 10:23:31.932016 25036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:31.953351 25036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:31.956367 25036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:31.956579 25036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:31.957253 25036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:31.958129 25036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:31.958174 25036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.958364 25036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:31.958441 25036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:31.967464 25036 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:36171
I20260929 10:23:31.967484 25164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:36171 every 8 connection(s)
I20260929 10:23:31.968391 25036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:31.970811 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25036
I20260929 10:23:31.970933 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:31.971253 21638 external_mini_cluster.cc:1428] Setting key 95383d1fb161bf5d861bb6f8daa40e0b
Sep 29 10:23:31 dist-test-slave-2rbl krb5kdc[24661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:31.982542 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:31.970289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:50059 (local address 127.21.33.190:35031)
0929 10:23:31.970438 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:31.970442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:31.971406 (+ 964us) server_negotiation.cc:407] Connection header received
0929 10:23:31.973003 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:31.973006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:31.973062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:31.973142 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:31.974315 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.974842 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.975997 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.976224 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.976338 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:31.978914 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:31.978918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:31.978920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:31.978944 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:31.980468 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.980990 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.980995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.980997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.981079 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:31.981456 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:31.981461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:31.981463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:31.981645 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:31.981763 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:31.982256 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 10:23:31.982380 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:23:31.983345 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:31.970729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35031 (local address 127.21.33.131:50059)
0929 10:23:31.971245 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:31.971308 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:31.972799 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:31.973338 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:31.973346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:31.973694 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:31.974157 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:31.974169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.974970 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.974975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:31.975865 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:31.975874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:31.976377 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:31.976382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:31.976560 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:31.977174 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:31.977231 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:31.978733 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:31.980594 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:31.980601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:31.980616 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:31.980884 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:31.981173 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:31.981177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:31.981179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:31.981359 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:31.981753 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:31.981759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:31.982016 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 10:23:31.982201 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 10:23:31.982537 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":150,"threads_started":1}
I20260929 10:23:31.984629 25165 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35031
I20260929 10:23:31.984882 25165 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:31.985455 25165 heartbeater.cc:511] Master 127.21.33.190:35031 requested a full tablet report, sending...
I20260929 10:23:31.986454 24706 ts_manager.cc:194] Registered new tserver with Master: 74d3d71711b542bbbc93b284ae3c3f0b (127.21.33.131:36171)
I20260929 10:23:31.987042 24706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:50059
I20260929 10:23:31.996985 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:23:32 dist-test-slave-2rbl krb5kdc[24661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677412, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 10:23:32 dist-test-slave-2rbl krb5kdc[24661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677412, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:32.024024 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:32.014730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.21.33.190:35031)
0929 10:23:32.014893 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:32.014899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:32.015067 (+ 168us) server_negotiation.cc:407] Connection header received
0929 10:23:32.015248 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:32.015251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:32.015293 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:32.015378 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:32.016098 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:32.016626 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:32.017543 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:32.017770 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:32.017884 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:32.020300 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:32.020310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:32.020313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:32.020338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:32.022126 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:32.022611 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:32.022615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:32.022617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:32.022717 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:32.023030 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:32.023035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:32.023037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:32.023290 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:32.023453 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:32.023717 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 10:23:32.023843 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 10:23:32.024488 24706 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42324
I20260929 10:23:32.025489 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24768
I20260929 10:23:32.031194 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24902
I20260929 10:23:32.036736 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25036
I20260929 10:23:32.042068 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 24674
2026-09-29T10:23:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2127 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:32 dist-test-slave-2rbl krb5kdc[25177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:32 dist-test-slave-2rbl krb5kdc[25177](info): set up 2 sockets
Sep 29 10:23:32 dist-test-slave-2rbl krb5kdc[25177](info): commencing operation
krb5kdc: starting...
W20260929 10:23:33.073932 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:33 dist-test-slave-2rbl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:33.194545 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:42619
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:42619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:33.304076 25190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:33.304376 25190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:33.304452 25190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:33.308729 25190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:33.308847 25190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:33.308892 25190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:33.308954 25190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:33.308996 25190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:33.314216 25190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:42619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:42619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:33.315265 25190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:33.316282 25190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:33.324965 25198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:33.324965 25195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:33.325098 25190 server_base.cc:1034] running on GCE node
W20260929 10:23:33.325289 25196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:33.325872 25190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:33.327004 25190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:33.328225 25190 hybrid_clock.cc:648] HybridClock initialized: now 1790677413328217 us; error 31 us; skew 500 ppm
Sep 29 10:23:33 dist-test-slave-2rbl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677413, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:33.331135 25190 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:33.331993 25190 webserver.cc:460] Webserver started at http://127.21.33.190:33365/ using document root <none> and password file <none>
I20260929 10:23:33.332553 25190 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:33.332643 25190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:33.332872 25190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:33.334911 25190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "2c6d17f1679648c3bf1108c07036eac8"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "73c24fb55fe353865e20a1d1f1d1eae1"
server_key_iv: "1c0d6d85a9833a19bd67ef296f0ec849"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.335455 25190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "2c6d17f1679648c3bf1108c07036eac8"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "73c24fb55fe353865e20a1d1f1d1eae1"
server_key_iv: "1c0d6d85a9833a19bd67ef296f0ec849"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.339035 25190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:33.341346 25205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:33.342549 25190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:33.342700 25190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "2c6d17f1679648c3bf1108c07036eac8"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "73c24fb55fe353865e20a1d1f1d1eae1"
server_key_iv: "1c0d6d85a9833a19bd67ef296f0ec849"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.342810 25190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:33.368633 25190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:33.372042 25190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:33.372260 25190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:33.379248 25190 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:42619
I20260929 10:23:33.379321 25258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:42619 every 8 connection(s)
I20260929 10:23:33.380215 25190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:33.380811 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25190
I20260929 10:23:33.380952 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:33.381316 21638 external_mini_cluster.cc:1428] Setting key 59e8659f75c979ac740a8bfbdbfbc0cb
I20260929 10:23:33.383287 25259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:33.389950 25259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8: Bootstrap starting.
Sep 29 10:23:33 dist-test-slave-2rbl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:33.392573 25259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:33.393498 25259 log.cc:824] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:33.395890 25259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8: No bootstrap required, opened a new log
I20260929 10:23:33.396957 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:33.382706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.21.33.190:42619)
0929 10:23:33.383121 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:33.383135 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:33.383171 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:23:33.383838 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:33.383854 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:33.384168 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:33.384450 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:33.385494 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.386908 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.388037 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.388494 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.388605 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:33.391336 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:33.391342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:33.391357 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:33.391387 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:33.393319 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:33.393762 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:33.393767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:33.393773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:33.393916 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:33.394229 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:33.394231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:33.394232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:33.394715 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:33.395591 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:33.395737 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 10:23:33.396030 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260929 10:23:33.399646 25259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d17f1679648c3bf1108c07036eac8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42619 } }
I20260929 10:23:33.399935 25259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:33.399999 25259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6d17f1679648c3bf1108c07036eac8, State: Initialized, Role: FOLLOWER
I20260929 10:23:33.400405 25259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d17f1679648c3bf1108c07036eac8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42619 } }
I20260929 10:23:33.400533 25259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:33.400578 25259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:33.400658 25259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:33.401749 25259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d17f1679648c3bf1108c07036eac8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42619 } }
I20260929 10:23:33.402113 25259 leader_election.cc:302] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c6d17f1679648c3bf1108c07036eac8; no voters:
I20260929 10:23:33.402428 25259 leader_election.cc:288] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:33.402565 25264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:33.402829 25264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6d17f1679648c3bf1108c07036eac8, State: Running, Role: LEADER
I20260929 10:23:33.403196 25264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d17f1679648c3bf1108c07036eac8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42619 } }
I20260929 10:23:33.403776 25259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:33.404896 25265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c6d17f1679648c3bf1108c07036eac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d17f1679648c3bf1108c07036eac8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42619 } } }
I20260929 10:23:33.405017 25265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:33.405366 25266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6d17f1679648c3bf1108c07036eac8. Latest consensus state: current_term: 1 leader_uuid: "2c6d17f1679648c3bf1108c07036eac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6d17f1679648c3bf1108c07036eac8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42619 } } }
I20260929 10:23:33.405462 25266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8 [sys.catalog]: This master's current role is: LEADER
W20260929 10:23:33.408643 25280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:23:33.408715 25280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:23:33.408815 25273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:33.409632 25273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:33.414634 25273 catalog_manager.cc:1347] Generated new cluster ID: e6be1115927b4445bbc5df4fa7a5ff55
I20260929 10:23:33.414705 25273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:33.434471 25273 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:33.435770 25273 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:33.457155 25273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c6d17f1679648c3bf1108c07036eac8: Generated new TSK 0
I20260929 10:23:33.457935 25273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:33.492509 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42619
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:33.626809 25285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:33.627059 25285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:33.627102 25285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:33.631094 25285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:33.631145 25285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:33.631202 25285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:33.636247 25285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:33.637269 25285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:33.638268 25285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:33.647668 25290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:33.647760 25285 server_base.cc:1034] running on GCE node
W20260929 10:23:33.647771 25291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:33.647927 25293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:33.648300 25285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:33.648955 25285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:33.650156 25285 hybrid_clock.cc:648] HybridClock initialized: now 1790677413650145 us; error 61 us; skew 500 ppm
Sep 29 10:23:33 dist-test-slave-2rbl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677413, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:33.653067 25285 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:33.654001 25285 webserver.cc:460] Webserver started at http://127.21.33.129:42669/ using document root <none> and password file <none>
I20260929 10:23:33.654544 25285 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:33.654632 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:33.654843 25285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:33.656793 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "66664f7352ee4a6897ad5b59ef3d862f"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "b4e20bcc65549502e4d41e2fffeb9287"
server_key_iv: "83affd739acd4c2800318c6864bed856"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.657403 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "66664f7352ee4a6897ad5b59ef3d862f"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "b4e20bcc65549502e4d41e2fffeb9287"
server_key_iv: "83affd739acd4c2800318c6864bed856"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.660923 25285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:23:33.663282 25300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:33.664382 25285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 10:23:33.664520 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "66664f7352ee4a6897ad5b59ef3d862f"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "b4e20bcc65549502e4d41e2fffeb9287"
server_key_iv: "83affd739acd4c2800318c6864bed856"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.664623 25285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:33.686784 25285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:33.689800 25285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:33.690006 25285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:33.690604 25285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:33.691427 25285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:33.691497 25285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:33.691567 25285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:33.691615 25285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:33.700254 25285 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:34711
I20260929 10:23:33.700286 25414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:34711 every 8 connection(s)
I20260929 10:23:33.701125 25285 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:33.709609 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25285
I20260929 10:23:33.709757 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:33.710054 21638 external_mini_cluster.cc:1428] Setting key 9ec821e64f7ebf28cefe3405d5c1b8ad
Sep 29 10:23:33 dist-test-slave-2rbl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677413, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:33.715538 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:33.702882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:41083 (local address 127.21.33.190:42619)
0929 10:23:33.703030 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:33.703034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:33.703905 (+ 871us) server_negotiation.cc:407] Connection header received
0929 10:23:33.705577 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:33.705580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:33.705639 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:33.705748 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:33.706904 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.707436 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.708572 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.708817 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.708929 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:33.711834 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:33.711837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:33.711839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:33.711862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:33.713491 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:33.713995 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:33.713999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:33.714001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:33.714089 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:33.714441 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:33.714444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:33.714446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:33.714629 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:33.714735 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:33.715233 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 10:23:33.715361 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:23:33.716334 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:33.703325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:42619 (local address 127.21.33.129:41083)
0929 10:23:33.703764 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:33.703807 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:33.705334 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:33.705919 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:33.705927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:33.706297 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:33.706763 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:33.706776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.707565 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.707568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:33.708447 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:33.708456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.708972 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.708976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:33.709152 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:33.709909 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:33.709925 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:33.711665 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:33.713606 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:33.713613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:33.713631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:33.713889 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:33.714180 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:33.714184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:33.714186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:33.714332 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:33.714743 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:33.714751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:33.714993 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 10:23:33.715182 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 10:23:33.715516 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260929 10:23:33.717561 25415 heartbeater.cc:347] Connected to a master server at 127.21.33.190:42619
I20260929 10:23:33.717828 25415 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:33.718451 25415 heartbeater.cc:511] Master 127.21.33.190:42619 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:23:33.720079 25222 ts_manager.cc:194] Registered new tserver with Master: 66664f7352ee4a6897ad5b59ef3d862f (127.21.33.129:34711)
I20260929 10:23:33.721479 25222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:41083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:33.736343 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42619
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:33.845005 25420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:33.845331 25420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:33.845381 25420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:33.849382 25420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:33.849436 25420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:33.849494 25420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:33.854784 25420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:33.855798 25420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:33.856751 25420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:23:33.866755 25420 server_base.cc:1034] running on GCE node
W20260929 10:23:33.866744 25425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:33.866748 25426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:33.866909 25428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:33.867427 25420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:33.868101 25420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:33.869271 25420 hybrid_clock.cc:648] HybridClock initialized: now 1790677413869249 us; error 62 us; skew 500 ppm
Sep 29 10:23:33 dist-test-slave-2rbl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677413, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:33.872083 25420 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:33.873019 25420 webserver.cc:460] Webserver started at http://127.21.33.130:39945/ using document root <none> and password file <none>
I20260929 10:23:33.873648 25420 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:33.873749 25420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:33.873991 25420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:33.876093 25420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "160cb80d6f974fa1ac41b0d3ca9e514e"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "1606840ffd9cc9bcc41db4961145b92a"
server_key_iv: "795bb538178a3e7d80efdeb059d83e2d"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.876675 25420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "160cb80d6f974fa1ac41b0d3ca9e514e"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "1606840ffd9cc9bcc41db4961145b92a"
server_key_iv: "795bb538178a3e7d80efdeb059d83e2d"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.880579 25420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:23:33.883358 25435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:33.884666 25420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:33.884820 25420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "160cb80d6f974fa1ac41b0d3ca9e514e"
format_stamp: "Formatted at 2026-09-29 10:23:33 on dist-test-slave-2rbl"
server_key: "1606840ffd9cc9bcc41db4961145b92a"
server_key_iv: "795bb538178a3e7d80efdeb059d83e2d"
server_key_version: "encryptionkey@0"
I20260929 10:23:33.884941 25420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:33.900005 25420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:33.903441 25420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:33.903646 25420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:33.904290 25420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:33.905287 25420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:33.905352 25420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:33.905417 25420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:33.905447 25420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:33.914218 25420 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:44195
I20260929 10:23:33.914249 25549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:44195 every 8 connection(s)
I20260929 10:23:33.915076 25420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:33.922497 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25420
I20260929 10:23:33.922626 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:33.922947 21638 external_mini_cluster.cc:1428] Setting key 3c2cae25d7b6e396ee379ebc3b6f9300
Sep 29 10:23:33 dist-test-slave-2rbl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677413, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:33.929430 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:33.916831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:36939 (local address 127.21.33.190:42619)
0929 10:23:33.916960 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:33.916964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:33.917889 (+ 925us) server_negotiation.cc:407] Connection header received
0929 10:23:33.919229 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:33.919232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:33.919277 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:33.919350 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:33.920499 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.921024 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.922301 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.922672 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.922804 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:33.925580 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:33.925587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:33.925593 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:33.925626 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:33.927179 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:33.927769 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:33.927774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:33.927775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:33.927861 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:33.928249 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:33.928253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:33.928255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:33.928453 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:33.928589 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:33.929083 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 10:23:33.929241 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:33.930208 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:33.917255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:42619 (local address 127.21.33.130:36939)
0929 10:23:33.917738 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:33.917781 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:33.919032 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:33.919509 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:33.919516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:33.919864 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:33.920359 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:33.920371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.921166 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.921169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:33.922157 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:33.922167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:33.922834 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:33.922837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:33.923005 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:33.923756 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:33.923772 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:33.925412 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:33.927292 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:33.927297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:33.927309 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:33.927638 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:33.927972 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:33.927975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:33.927980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:33.928135 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:33.928571 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:33.928577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:33.928826 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 10:23:33.929023 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 10:23:33.929388 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":106,"threads_started":1}
I20260929 10:23:33.931497 25550 heartbeater.cc:347] Connected to a master server at 127.21.33.190:42619
I20260929 10:23:33.931742 25550 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:33.932225 25550 heartbeater.cc:511] Master 127.21.33.190:42619 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:33.933341 25222 ts_manager.cc:194] Registered new tserver with Master: 160cb80d6f974fa1ac41b0d3ca9e514e (127.21.33.130:44195)
I20260929 10:23:33.933966 25222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:36939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:33.953154 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42619
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:34.062119 25555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:34.062377 25555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:34.062445 25555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:34.066475 25555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:34.066534 25555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:34.066620 25555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:34.071992 25555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:34.073060 25555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:34.074110 25555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:34.083719 25560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:34.083719 25563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:34.083786 25561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:34.083889 25555 server_base.cc:1034] running on GCE node
I20260929 10:23:34.084409 25555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:34.085107 25555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:34.086315 25555 hybrid_clock.cc:648] HybridClock initialized: now 1790677414086300 us; error 34 us; skew 500 ppm
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677414, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:34.089342 25555 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:34.090180 25555 webserver.cc:460] Webserver started at http://127.21.33.131:46347/ using document root <none> and password file <none>
I20260929 10:23:34.090709 25555 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:34.090767 25555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:34.090936 25555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:34.093276 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "ec3f0a463226473f9cdcabc64c57fd61"
format_stamp: "Formatted at 2026-09-29 10:23:34 on dist-test-slave-2rbl"
server_key: "88bfb51a4cb04f5afe5ae7367b194c2e"
server_key_iv: "f14aff4039a4a55cb14a39a7d76a8e73"
server_key_version: "encryptionkey@0"
I20260929 10:23:34.093956 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "ec3f0a463226473f9cdcabc64c57fd61"
format_stamp: "Formatted at 2026-09-29 10:23:34 on dist-test-slave-2rbl"
server_key: "88bfb51a4cb04f5afe5ae7367b194c2e"
server_key_iv: "f14aff4039a4a55cb14a39a7d76a8e73"
server_key_version: "encryptionkey@0"
I20260929 10:23:34.097671 25555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:23:34.100057 25570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:34.101348 25555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:34.101475 25555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "ec3f0a463226473f9cdcabc64c57fd61"
format_stamp: "Formatted at 2026-09-29 10:23:34 on dist-test-slave-2rbl"
server_key: "88bfb51a4cb04f5afe5ae7367b194c2e"
server_key_iv: "f14aff4039a4a55cb14a39a7d76a8e73"
server_key_version: "encryptionkey@0"
I20260929 10:23:34.101585 25555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:34.114595 25555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:34.117451 25555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:34.117648 25555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:34.118227 25555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:34.119040 25555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:34.119113 25555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:34.119180 25555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:34.119234 25555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:34.127995 25555 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:38225
I20260929 10:23:34.128021 25684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:38225 every 8 connection(s)
I20260929 10:23:34.128841 25555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:34.129418 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25555
I20260929 10:23:34.129549 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:34.129815 21638 external_mini_cluster.cc:1428] Setting key a2959f30669a6570d470cd1c51336604
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677414, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:34.142968 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:34.130799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:36307 (local address 127.21.33.190:42619)
0929 10:23:34.130931 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:34.130934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:34.132099 (+ 1165us) server_negotiation.cc:407] Connection header received
0929 10:23:34.133520 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:34.133522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:34.133565 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:34.133633 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:34.134692 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.135206 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.136317 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.136540 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.136637 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:34.138974 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:34.138978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:34.138983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:34.139010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:34.140817 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:34.141419 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:34.141423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:34.141425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:34.141509 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:34.141869 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:34.141872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:34.141874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:34.142065 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:34.142170 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:34.142678 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 10:23:34.142799 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:23:34.143729 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:34.131296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:42619 (local address 127.21.33.131:36307)
0929 10:23:34.131887 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:34.132004 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:34.133341 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:34.133760 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:34.133766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:34.134112 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:34.134567 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:34.134579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.135314 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.135317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:34.136186 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:34.136195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.136674 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.136677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:34.136814 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:34.137357 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:34.137373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:34.138807 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:34.140954 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:34.140959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:34.140973 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:34.141310 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:34.141609 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:34.141613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:34.141615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:34.141765 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:34.142180 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:34.142188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:34.142434 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 10:23:34.142615 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 10:23:34.142927 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
I20260929 10:23:34.144795 25685 heartbeater.cc:347] Connected to a master server at 127.21.33.190:42619
I20260929 10:23:34.145023 25685 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:34.145601 25685 heartbeater.cc:511] Master 127.21.33.190:42619 requested a full tablet report, sending...
I20260929 10:23:34.146582 25222 ts_manager.cc:194] Registered new tserver with Master: ec3f0a463226473f9cdcabc64c57fd61 (127.21.33.131:38225)
I20260929 10:23:34.147156 25222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:36307
I20260929 10:23:34.155629 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:34.181782 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:34.172186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.21.33.129:34711)
0929 10:23:34.172457 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:34.172472 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:34.172496 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:23:34.172562 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:34.172567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:34.172794 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:34.172911 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:34.173647 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.174434 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.175248 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.175660 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.175751 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:34.178201 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:34.178210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:34.178223 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:34.178246 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:34.179941 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:34.180393 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:34.180399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:34.180400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:34.180476 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:34.180759 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:34.180764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:34.180765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:34.181002 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:34.181257 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:34.181444 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 10:23:34.181608 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:34.192348 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:34.184594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.21.33.190:42619)
0929 10:23:34.184748 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:34.184751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:34.184765 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:34.184889 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:34.184891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:34.184932 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:34.184998 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:34.185609 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.186113 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.186897 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:34.187098 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:34.187172 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:34.189615 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:34.189618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:34.189620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:34.189644 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:34.190887 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:34.191259 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:34.191262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:34.191263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:34.191333 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:34.191556 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:34.191558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:34.191559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:34.191767 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:34.191868 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:34.192051 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 10:23:34.192141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 10:23:34.192800 25222 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59984
I20260929 10:23:34.193544 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25285
I20260929 10:23:34.199219 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25420
I20260929 10:23:34.205973 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25555
I20260929 10:23:34.211053 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25190
2026-09-29T10:23:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2168 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25697](info): set up 2 sockets
Sep 29 10:23:34 dist-test-slave-2rbl krb5kdc[25697](info): commencing operation
krb5kdc: starting...
W20260929 10:23:35.269812 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:35 dist-test-slave-2rbl krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:35.391811 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39667
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:39667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:35.502923 25710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:35.503225 25710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:35.503304 25710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:35.507328 25710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:35.507409 25710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:35.507447 25710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:35.507505 25710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:35.507543 25710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:35.512573 25710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:39667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:35.513636 25710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:35.514668 25710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:35.523021 25716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:35.523036 25718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:35.523083 25710 server_base.cc:1034] running on GCE node
W20260929 10:23:35.523039 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:35.523840 25710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:35.524914 25710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:35.526101 25710 hybrid_clock.cc:648] HybridClock initialized: now 1790677415526061 us; error 53 us; skew 500 ppm
Sep 29 10:23:35 dist-test-slave-2rbl krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677415, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:35.528807 25710 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:35.529580 25710 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:35.529644 25710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:35.529805 25710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:35.531728 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "37e38cd9eb7749b38ddb77f1c470077c"
format_stamp: "Formatted at 2026-09-29 10:23:35 on dist-test-slave-2rbl"
server_key: "6bdde44f73e9bef6cd818cdf28702155"
server_key_iv: "57001f22e2693acb5dc353d5823fa4cc"
server_key_version: "encryptionkey@0"
I20260929 10:23:35.532235 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "37e38cd9eb7749b38ddb77f1c470077c"
format_stamp: "Formatted at 2026-09-29 10:23:35 on dist-test-slave-2rbl"
server_key: "6bdde44f73e9bef6cd818cdf28702155"
server_key_iv: "57001f22e2693acb5dc353d5823fa4cc"
server_key_version: "encryptionkey@0"
I20260929 10:23:35.535741 25710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:23:35.537994 25724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:35.539162 25710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:35.539291 25710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "37e38cd9eb7749b38ddb77f1c470077c"
format_stamp: "Formatted at 2026-09-29 10:23:35 on dist-test-slave-2rbl"
server_key: "6bdde44f73e9bef6cd818cdf28702155"
server_key_iv: "57001f22e2693acb5dc353d5823fa4cc"
server_key_version: "encryptionkey@0"
I20260929 10:23:35.539394 25710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:35.562276 25710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:35.565688 25710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:35.565891 25710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:35.572835 25710 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:39667
I20260929 10:23:35.573086 25776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:39667 every 8 connection(s)
I20260929 10:23:35.573513 25710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:35.576262 25777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:35.578504 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25710
I20260929 10:23:35.578621 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:35.578924 21638 external_mini_cluster.cc:1428] Setting key 41f7ce6559c394dce7aba6f5025a0b7f
I20260929 10:23:35.582784 25777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c: Bootstrap starting.
I20260929 10:23:35.585479 25777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:35.586341 25777 log.cc:824] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:35.588459 25777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c: No bootstrap required, opened a new log
Sep 29 10:23:35 dist-test-slave-2rbl krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:35.592092 25777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e38cd9eb7749b38ddb77f1c470077c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39667 } }
I20260929 10:23:35.592372 25777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:35.592458 25777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e38cd9eb7749b38ddb77f1c470077c, State: Initialized, Role: FOLLOWER
I20260929 10:23:35.592939 25777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e38cd9eb7749b38ddb77f1c470077c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39667 } }
I20260929 10:23:35.593096 25777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:35.593163 25777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:35.593276 25777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:35.594435 25777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e38cd9eb7749b38ddb77f1c470077c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39667 } }
I20260929 10:23:35.594795 25777 leader_election.cc:302] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e38cd9eb7749b38ddb77f1c470077c; no voters:
I20260929 10:23:35.595121 25777 leader_election.cc:288] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:35.595157 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:35.580471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.21.33.190:39667)
0929 10:23:35.580934 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:35.580953 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:35.580995 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:23:35.581831 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:35.581856 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:35.582274 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:35.582648 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:35.583444 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:35.584894 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:35.585965 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:35.586657 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:35.586825 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:35.589585 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:35.589590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:35.589603 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:35.589630 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:35.591785 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:35.592287 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:35.592291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:35.592295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:35.592479 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:35.592823 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:35.592826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:35.592828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:35.593403 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:35.593618 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:35.593846 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 10:23:35.594153 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":145,"threads_started":1}
I20260929 10:23:35.595254 25782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:35.595475 25782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [term 1 LEADER]: Becoming Leader. State: Replica: 37e38cd9eb7749b38ddb77f1c470077c, State: Running, Role: LEADER
I20260929 10:23:35.595780 25782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e38cd9eb7749b38ddb77f1c470077c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39667 } }
I20260929 10:23:35.596349 25777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:35.597002 25784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e38cd9eb7749b38ddb77f1c470077c. Latest consensus state: current_term: 1 leader_uuid: "37e38cd9eb7749b38ddb77f1c470077c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e38cd9eb7749b38ddb77f1c470077c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39667 } } }
I20260929 10:23:35.597119 25784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:35.596983 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37e38cd9eb7749b38ddb77f1c470077c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e38cd9eb7749b38ddb77f1c470077c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39667 } } }
I20260929 10:23:35.597379 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c [sys.catalog]: This master's current role is: LEADER
W20260929 10:23:35.600760 25798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:23:35.600826 25798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:23:35.600911 25789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:35.601653 25789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:35.607647 25789 catalog_manager.cc:1347] Generated new cluster ID: d037572f05f24db08684ef16fad30de9
I20260929 10:23:35.607725 25789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:35.624527 25789 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:35.625566 25789 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:35.632793 25789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c: Generated new TSK 0
I20260929 10:23:35.633496 25789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:35.668757 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39667
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:35.780772 25804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:35.781065 25804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:35.781147 25804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:35.785145 25804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:35.785264 25804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:35.785362 25804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:35.790524 25804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:35.791579 25804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:35.792616 25804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:35.802191 25809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:35.802214 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:35.802191 25813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:35.802531 25804 server_base.cc:1034] running on GCE node
I20260929 10:23:35.802979 25804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:35.803608 25804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:35.804791 25804 hybrid_clock.cc:648] HybridClock initialized: now 1790677415804781 us; error 33 us; skew 500 ppm
Sep 29 10:23:35 dist-test-slave-2rbl krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677415, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:35.807546 25804 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:35.808300 25804 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:35.808398 25804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:35.808630 25804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:35.810671 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "f642f32582fb4ae18e15ce38391997f4"
format_stamp: "Formatted at 2026-09-29 10:23:35 on dist-test-slave-2rbl"
server_key: "4b6584fad23fb9e42cd3918ecda78e08"
server_key_iv: "a7b886297a467cecae4ec864bdb193b9"
server_key_version: "encryptionkey@0"
I20260929 10:23:35.811219 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "f642f32582fb4ae18e15ce38391997f4"
format_stamp: "Formatted at 2026-09-29 10:23:35 on dist-test-slave-2rbl"
server_key: "4b6584fad23fb9e42cd3918ecda78e08"
server_key_iv: "a7b886297a467cecae4ec864bdb193b9"
server_key_version: "encryptionkey@0"
I20260929 10:23:35.814798 25804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:35.817009 25819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:35.818397 25804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:35.818495 25804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "f642f32582fb4ae18e15ce38391997f4"
format_stamp: "Formatted at 2026-09-29 10:23:35 on dist-test-slave-2rbl"
server_key: "4b6584fad23fb9e42cd3918ecda78e08"
server_key_iv: "a7b886297a467cecae4ec864bdb193b9"
server_key_version: "encryptionkey@0"
I20260929 10:23:35.818584 25804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:35.862361 25804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:35.865480 25804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:35.865636 25804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:35.866161 25804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:35.866937 25804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:35.866982 25804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:35.867023 25804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:35.867038 25804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:35.875699 25804 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:37751
I20260929 10:23:35.875825 25933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:37751 every 8 connection(s)
I20260929 10:23:35.876358 25804 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:23:35 dist-test-slave-2rbl krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677415, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:35.886040 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25804
I20260929 10:23:35.886186 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:35.886529 21638 external_mini_cluster.cc:1428] Setting key 614faed0f81593ce06f9bba4e78da422
I20260929 10:23:35.890108 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:35.878124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:58747 (local address 127.21.33.190:39667)
0929 10:23:35.878305 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:35.878321 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:35.879224 (+ 903us) server_negotiation.cc:407] Connection header received
0929 10:23:35.880457 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:35.880460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:35.880508 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:35.880583 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:35.881843 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:35.882384 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:35.883501 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:35.883730 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:35.883816 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:35.886430 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:35.886433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:35.886435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:35.886462 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:35.888037 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:35.888542 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:35.888547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:35.888549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:35.888631 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:35.888972 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:35.888977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:35.888979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:35.889156 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:35.889342 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:35.889836 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 10:23:35.889955 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 10:23:35.890949 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:35.878546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39667 (local address 127.21.33.129:58747)
0929 10:23:35.879053 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:35.879098 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:35.880255 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:35.880772 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:35.880779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:35.881132 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:35.881708 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:35.881722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:35.882495 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:35.882497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:35.883382 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:35.883391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:35.883868 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:35.883871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:35.884013 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:35.884573 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:35.884588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:35.886256 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:35.888156 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:35.888162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:35.888178 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:35.888439 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:35.888728 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:35.888731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:35.888733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:35.888879 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:35.889342 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:35.889350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:35.889600 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 10:23:35.889798 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 10:23:35.890122 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
I20260929 10:23:35.892313 25934 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39667
I20260929 10:23:35.892555 25934 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:35.892997 25934 heartbeater.cc:511] Master 127.21.33.190:39667 requested a full tablet report, sending...
I20260929 10:23:35.894564 25741 ts_manager.cc:194] Registered new tserver with Master: f642f32582fb4ae18e15ce38391997f4 (127.21.33.129:37751)
I20260929 10:23:35.895658 25741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:58747
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:35.914283 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39667
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:36.023775 25939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:36.024021 25939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:36.024063 25939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:36.028033 25939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:36.028090 25939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:36.028146 25939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:36.033238 25939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:36.034250 25939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:36.035283 25939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:36.044739 25947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:36.044776 25944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:36.044793 25945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:36.045027 25939 server_base.cc:1034] running on GCE node
I20260929 10:23:36.045460 25939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:36.046082 25939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:36.047240 25939 hybrid_clock.cc:648] HybridClock initialized: now 1790677416047228 us; error 29 us; skew 500 ppm
Sep 29 10:23:36 dist-test-slave-2rbl krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677416, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:36.049928 25939 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:36.050661 25939 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:36.050724 25939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:36.050894 25939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:36.052837 25939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "3955a338231e4479a336cc12ffa54eff"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "eef0f99ea5b28826642c796158a95d1d"
server_key_iv: "d4da54ca0088c00220f739d931ea037c"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.053409 25939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "3955a338231e4479a336cc12ffa54eff"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "eef0f99ea5b28826642c796158a95d1d"
server_key_iv: "d4da54ca0088c00220f739d931ea037c"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.056922 25939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 10:23:36.059355 25953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.060575 25939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:36.060704 25939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "3955a338231e4479a336cc12ffa54eff"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "eef0f99ea5b28826642c796158a95d1d"
server_key_iv: "d4da54ca0088c00220f739d931ea037c"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.060824 25939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:36.077126 25939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:36.080289 25939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:36.080520 25939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:36.081121 25939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:36.081969 25939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:36.082042 25939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.082113 25939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:36.082156 25939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.091526 25939 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:44325
I20260929 10:23:36.091557 26067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:44325 every 8 connection(s)
I20260929 10:23:36.092118 25939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:36.100487 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 25939
I20260929 10:23:36.100610 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:36.100880 21638 external_mini_cluster.cc:1428] Setting key c4dad3b48f98a20c4e06534b72837737
Sep 29 10:23:36 dist-test-slave-2rbl krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677416, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:36.106343 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.093878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:37531 (local address 127.21.33.190:39667)
0929 10:23:36.094034 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.094038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.094958 (+ 920us) server_negotiation.cc:407] Connection header received
0929 10:23:36.096319 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.096323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.096377 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.096481 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:36.097877 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.098436 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.099578 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.099809 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.099919 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.102456 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:36.102459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:36.102462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:36.102488 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:36.104013 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:36.104664 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:36.104669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:36.104671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:36.104767 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:36.105082 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:36.105088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:36.105091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:36.105384 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:36.105499 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.105953 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.106090 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:23:36.106987 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.094361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39667 (local address 127.21.33.130:37531)
0929 10:23:36.094816 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.094857 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:36.096098 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:36.096650 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:36.096662 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:36.097047 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:36.097737 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.097753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.098568 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.098572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.099452 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.099461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.099962 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.099966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.100137 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.100754 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:36.100769 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:36.102302 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:36.104131 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:36.104140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:36.104158 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:36.104550 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:36.104863 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:36.104865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:36.104866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:36.104994 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:36.105485 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:36.105490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:36.105708 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 10:23:36.105863 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 10:23:36.106099 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":104,"threads_started":1}
I20260929 10:23:36.108172 26068 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39667
I20260929 10:23:36.108438 26068 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:36.108978 26068 heartbeater.cc:511] Master 127.21.33.190:39667 requested a full tablet report, sending...
I20260929 10:23:36.110083 25741 ts_manager.cc:194] Registered new tserver with Master: 3955a338231e4479a336cc12ffa54eff (127.21.33.130:44325)
I20260929 10:23:36.110637 25741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:37531
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:36.130477 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39667
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:36.240450 26073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:36.240861 26073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:36.240926 26073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:36.245280 26073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:36.245337 26073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:36.245399 26073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:36.250947 26073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:36.251961 26073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:36.252997 26073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:36.262835 26081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:36.262841 26079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:36.262848 26078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:36.262895 26073 server_base.cc:1034] running on GCE node
I20260929 10:23:36.263597 26073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:36.264288 26073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:36.265473 26073 hybrid_clock.cc:648] HybridClock initialized: now 1790677416265437 us; error 52 us; skew 500 ppm
Sep 29 10:23:36 dist-test-slave-2rbl krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677416, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:36.268256 26073 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:36.269045 26073 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:36.269141 26073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:36.269398 26073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:36.271432 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "9e004dbb22b74eec82d9356732b0cc79"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "0496da331ac80dbe1451cc26028ad011"
server_key_iv: "4c73717ca5245513bf45391951619265"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.272022 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "9e004dbb22b74eec82d9356732b0cc79"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "0496da331ac80dbe1451cc26028ad011"
server_key_iv: "4c73717ca5245513bf45391951619265"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.275813 26073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:23:36.278337 26087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.279547 26073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:36.279681 26073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "9e004dbb22b74eec82d9356732b0cc79"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "0496da331ac80dbe1451cc26028ad011"
server_key_iv: "4c73717ca5245513bf45391951619265"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.279811 26073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:36.290927 26073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:36.293573 26073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:36.293769 26073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:36.294309 26073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:36.295102 26073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:36.295171 26073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.295233 26073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:36.295287 26073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.303857 26073 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:40047
I20260929 10:23:36.303881 26201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:40047 every 8 connection(s)
I20260929 10:23:36.304386 26073 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:36.307096 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26073
I20260929 10:23:36.307243 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:36.307521 21638 external_mini_cluster.cc:1428] Setting key 2ebcf01930e227943e7be60c28a0fa3b
Sep 29 10:23:36 dist-test-slave-2rbl krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677416, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:36.317332 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.305987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:49085 (local address 127.21.33.190:39667)
0929 10:23:36.306132 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.306136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.306910 (+ 774us) server_negotiation.cc:407] Connection header received
0929 10:23:36.308408 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.308411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.308464 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.308542 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:36.309924 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.310445 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.311492 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.311693 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.311767 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.314116 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:36.314119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:36.314121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:36.314142 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:36.315390 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:36.315837 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:36.315839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:36.315840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:36.315908 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:36.316291 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:36.316293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:36.316294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:36.316451 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:36.316524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.317051 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.317139 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:23:36.318113 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.306336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39667 (local address 127.21.33.131:49085)
0929 10:23:36.306761 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.306801 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:36.308178 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:36.308743 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:36.308750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:36.309170 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:36.309799 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.309815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.310544 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.310546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.311388 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.311397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.311810 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.311813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.311941 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.312537 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:36.312553 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:36.313971 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:36.315495 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:36.315499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:36.315509 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:36.315755 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:36.316019 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:36.316021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:36.316023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:36.316201 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:36.316576 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:36.316582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:36.316851 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 10:23:36.317008 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 10:23:36.317311 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260929 10:23:36.319187 26202 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39667
I20260929 10:23:36.319424 26202 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:36.319844 26202 heartbeater.cc:511] Master 127.21.33.190:39667 requested a full tablet report, sending...
I20260929 10:23:36.320926 25741 ts_manager.cc:194] Registered new tserver with Master: 9e004dbb22b74eec82d9356732b0cc79 (127.21.33.131:40047)
I20260929 10:23:36.321302 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:23:36.321621 25741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:49085
I20260929 10:23:36.330323 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.323484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.21.33.190:39667)
0929 10:23:36.323629 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.323633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.323793 (+ 160us) server_negotiation.cc:407] Connection header received
0929 10:23:36.323946 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.323949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.323988 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.324073 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:36.324762 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.325347 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.326230 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.326428 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.326502 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.327810 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:36.327813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:36.327815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:36.327834 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:36.328994 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:36.329419 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:36.329421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:36.329422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:36.329491 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:36.329696 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:36.329698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:36.329699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:36.329871 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:36.329964 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.330126 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.330217 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:23:36.333168 25741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:36.336055 25741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:36.350250 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.344746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37751 (local address 127.0.0.1:60238)
0929 10:23:36.345085 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.345097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:36.345178 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:36.346258 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:36.346262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:36.346280 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:36.346354 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.346361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.347585 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.347591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.348586 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.348595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.349668 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.349672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.349790 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.349833 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 10:23:36.349949 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 10:23:36.350086 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":87,"threads_started":1}
I20260929 10:23:36.350267 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.345376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.21.33.129:37751)
0929 10:23:36.345776 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.345781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.345797 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:36.345887 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.345891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.346074 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.346262 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:36.346454 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.347425 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.348719 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.349532 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.349650 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.349811 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.349996 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.350091 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
I20260929 10:23:36.350575 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.344955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.21.33.131:40047)
0929 10:23:36.345492 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.345529 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.345553 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:23:36.345618 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.345622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.345805 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.345957 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:36.346171 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.347011 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.348888 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.349680 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.349775 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.349887 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.350328 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.350446 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":159,"threads_started":1}
I20260929 10:23:36.350683 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.344018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:44325 (local address 127.0.0.1:47070)
0929 10:23:36.344360 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.344408 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:36.344520 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:36.345588 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:36.345593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:36.345616 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:36.345751 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.345758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.347186 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.347189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.348754 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.348766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.350279 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.350284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.350382 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.350408 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:23:36.350482 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:23:36.350555 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":116,"thread_start_us":93,"threads_started":1}
I20260929 10:23:36.350683 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.344109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:40047 (local address 127.0.0.1:60084)
0929 10:23:36.344723 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.344737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:36.344818 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:36.345935 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:36.345938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:36.345955 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:36.346049 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.346056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.347143 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.347150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.348763 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.348769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.349979 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.349982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.350069 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.350092 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:23:36.350163 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:23:36.350245 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":125,"thread_start_us":93,"threads_started":1}
I20260929 10:23:36.351289 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.344113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.21.33.130:44325)
0929 10:23:36.345051 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.345064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.345089 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:23:36.345168 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.345173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.345409 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.345578 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:36.345878 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.347054 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.349305 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.350000 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.350094 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.350215 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.351026 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.351125 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":112,"threads_started":1}
I20260929 10:23:36.353580 26135 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8c9b520f97455a93fb943b0613f7b1 (DEFAULT_TABLE table=test-table [id=b3cb7413110c42ad8fa3f32d1b643677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:36.353670 25867 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8c9b520f97455a93fb943b0613f7b1 (DEFAULT_TABLE table=test-table [id=b3cb7413110c42ad8fa3f32d1b643677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:36.353667 26001 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8c9b520f97455a93fb943b0613f7b1 (DEFAULT_TABLE table=test-table [id=b3cb7413110c42ad8fa3f32d1b643677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:36.354627 26135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8c9b520f97455a93fb943b0613f7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:36.354704 25867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8c9b520f97455a93fb943b0613f7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:36.354704 26001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8c9b520f97455a93fb943b0613f7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:36.360884 26219 tablet_bootstrap.cc:492] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Bootstrap starting.
I20260929 10:23:36.362411 26220 tablet_bootstrap.cc:492] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Bootstrap starting.
I20260929 10:23:36.363175 26219 tablet_bootstrap.cc:654] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:36.364261 26219 log.cc:824] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:36.364287 26220 tablet_bootstrap.cc:654] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:36.365420 26220 log.cc:824] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:36.365658 26221 tablet_bootstrap.cc:492] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: Bootstrap starting.
I20260929 10:23:36.366648 26219 tablet_bootstrap.cc:492] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: No bootstrap required, opened a new log
I20260929 10:23:36.366848 26219 ts_tablet_manager.cc:1397] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:23:36.367352 26220 tablet_bootstrap.cc:492] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: No bootstrap required, opened a new log
I20260929 10:23:36.367527 26220 ts_tablet_manager.cc:1397] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:36.367622 26221 tablet_bootstrap.cc:654] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:36.368579 26221 log.cc:824] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:36.370167 26221 tablet_bootstrap.cc:492] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: No bootstrap required, opened a new log
I20260929 10:23:36.370374 26221 ts_tablet_manager.cc:1397] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:36.370791 26219 raft_consensus.cc:348] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.371078 26219 raft_consensus.cc:374] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:36.371136 26219 raft_consensus.cc:729] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e004dbb22b74eec82d9356732b0cc79, State: Initialized, Role: FOLLOWER
I20260929 10:23:36.371323 26220 raft_consensus.cc:348] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.371578 26220 raft_consensus.cc:374] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:36.371635 26220 raft_consensus.cc:729] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3955a338231e4479a336cc12ffa54eff, State: Initialized, Role: FOLLOWER
I20260929 10:23:36.371594 26219 consensus_queue.cc:260] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.372303 26202 heartbeater.cc:503] Master 127.21.33.190:39667 was elected leader, sending a full tablet report...
I20260929 10:23:36.372082 26220 consensus_queue.cc:260] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.372582 26219 ts_tablet_manager.cc:1428] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:23:36.373564 26220 ts_tablet_manager.cc:1428] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:23:36.373693 26068 heartbeater.cc:503] Master 127.21.33.190:39667 was elected leader, sending a full tablet report...
I20260929 10:23:36.374037 26221 raft_consensus.cc:348] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.374375 26221 raft_consensus.cc:374] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:36.374446 26221 raft_consensus.cc:729] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f642f32582fb4ae18e15ce38391997f4, State: Initialized, Role: FOLLOWER
I20260929 10:23:36.374929 26221 consensus_queue.cc:260] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.375813 25934 heartbeater.cc:503] Master 127.21.33.190:39667 was elected leader, sending a full tablet report...
I20260929 10:23:36.375931 26221 ts_tablet_manager.cc:1428] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 10:23:36.377347 25935 tablet.cc:2369] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:36.484690 26226 raft_consensus.cc:482] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:36.484931 26226 raft_consensus.cc:504] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.486192 26226 leader_election.cc:288] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e004dbb22b74eec82d9356732b0cc79 (127.21.33.131:40047), f642f32582fb4ae18e15ce38391997f4 (127.21.33.129:37751)
I20260929 10:23:36.490288 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.486506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:40047 (local address 127.21.33.130:35555)
0929 10:23:36.486604 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.486621 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:36.486721 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:36.487055 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:36.487057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:36.487075 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:36.487238 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.487245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.488129 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.488132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.489008 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.489017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.489819 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.489823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.489936 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.489996 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 10:23:36.490072 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:23:36.490128 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":25}
I20260929 10:23:36.490336 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.486587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:35555 (local address 127.21.33.131:40047)
0929 10:23:36.486700 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.486705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.486724 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:23:36.486876 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.486881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.486932 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.487047 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:36.487352 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.488018 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.489158 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.489679 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.489777 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.489818 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.490113 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.490210 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:23:36.490407 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.486577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:53091 (local address 127.21.33.129:37751)
0929 10:23:36.486700 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.486705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.487121 (+ 416us) server_negotiation.cc:407] Connection header received
0929 10:23:36.487283 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.487289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.487349 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.487421 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:36.487689 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.488223 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.489372 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.489812 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.489929 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.489990 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.490210 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.490288 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:36.490825 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.486506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37751 (local address 127.21.33.130:53091)
0929 10:23:36.487010 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.487025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:36.487158 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:36.487458 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:36.487461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:36.487479 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:36.487583 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.487590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.488368 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.488371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.489253 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:36.489263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.489938 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.489941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:36.490035 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.490083 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 10:23:36.490179 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:23:36.490243 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":110,"spinlock_wait_cycles":1792,"thread_start_us":123,"threads_started":1}
I20260929 10:23:36.490983 26155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8c9b520f97455a93fb943b0613f7b1" candidate_uuid: "3955a338231e4479a336cc12ffa54eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e004dbb22b74eec82d9356732b0cc79" is_pre_election: true
I20260929 10:23:36.491321 26155 raft_consensus.cc:2454] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3955a338231e4479a336cc12ffa54eff in term 0.
I20260929 10:23:36.491462 25887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8c9b520f97455a93fb943b0613f7b1" candidate_uuid: "3955a338231e4479a336cc12ffa54eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f642f32582fb4ae18e15ce38391997f4" is_pre_election: true
I20260929 10:23:36.491768 25887 raft_consensus.cc:2454] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3955a338231e4479a336cc12ffa54eff in term 0.
I20260929 10:23:36.491830 25957 leader_election.cc:302] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3955a338231e4479a336cc12ffa54eff, 9e004dbb22b74eec82d9356732b0cc79; no voters:
I20260929 10:23:36.492141 26226 raft_consensus.cc:2781] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:36.492236 26226 raft_consensus.cc:482] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:36.492281 26226 raft_consensus.cc:3037] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:36.493381 26226 raft_consensus.cc:504] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.493708 26226 leader_election.cc:288] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [CANDIDATE]: Term 1 election: Requested vote from peers 9e004dbb22b74eec82d9356732b0cc79 (127.21.33.131:40047), f642f32582fb4ae18e15ce38391997f4 (127.21.33.129:37751)
I20260929 10:23:36.494022 26155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8c9b520f97455a93fb943b0613f7b1" candidate_uuid: "3955a338231e4479a336cc12ffa54eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e004dbb22b74eec82d9356732b0cc79"
I20260929 10:23:36.494033 25887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8c9b520f97455a93fb943b0613f7b1" candidate_uuid: "3955a338231e4479a336cc12ffa54eff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f642f32582fb4ae18e15ce38391997f4"
I20260929 10:23:36.494129 25887 raft_consensus.cc:3037] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:36.494174 26155 raft_consensus.cc:3037] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:36.495451 25887 raft_consensus.cc:2454] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3955a338231e4479a336cc12ffa54eff in term 1.
I20260929 10:23:36.495545 26155 raft_consensus.cc:2454] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3955a338231e4479a336cc12ffa54eff in term 1.
I20260929 10:23:36.495777 25956 leader_election.cc:302] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3955a338231e4479a336cc12ffa54eff, f642f32582fb4ae18e15ce38391997f4; no voters:
I20260929 10:23:36.496009 26226 raft_consensus.cc:2781] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:36.496230 26226 raft_consensus.cc:686] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 1 LEADER]: Becoming Leader. State: Replica: 3955a338231e4479a336cc12ffa54eff, State: Running, Role: LEADER
I20260929 10:23:36.496569 26226 consensus_queue.cc:237] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } }
I20260929 10:23:36.499717 25739 catalog_manager.cc:5495] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3955a338231e4479a336cc12ffa54eff (127.21.33.130). New cstate: current_term: 1 leader_uuid: "3955a338231e4479a336cc12ffa54eff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3955a338231e4479a336cc12ffa54eff" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:36.539899 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.536579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.21.33.130:44325)
0929 10:23:36.536714 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.536718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.536730 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:36.536832 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.536834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.536889 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.536977 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:36.537246 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.537759 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.538570 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:36.538787 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:36.538866 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:36.539146 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:36.539554 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:36.539647 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.539723 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.539773 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:23:36.546046 25887 raft_consensus.cc:1261] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4 [term 1 FOLLOWER]: Refusing update from remote peer 3955a338231e4479a336cc12ffa54eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:36.546046 26155 raft_consensus.cc:1261] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 1 FOLLOWER]: Refusing update from remote peer 3955a338231e4479a336cc12ffa54eff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:36.546775 26229 consensus_queue.cc:1035] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f642f32582fb4ae18e15ce38391997f4" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:36.546933 26226 consensus_queue.cc:1035] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e004dbb22b74eec82d9356732b0cc79" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:23:36.555167 26203 tablet.cc:2369] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:36.557716 26233 mvcc.cc:205] Tried to move back new op lower bound from 7334614698164969472 to 7334614697973080064. Current Snapshot: MvccSnapshot[applied={T|T < 7334614698164969472}]
I20260929 10:23:36.562922 25739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:36.563100 25739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:36.565603 25739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37e38cd9eb7749b38ddb77f1c470077c: Sending DeleteTablet for 3 replicas of tablet 8a8c9b520f97455a93fb943b0613f7b1
I20260929 10:23:36.566321 26001 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8c9b520f97455a93fb943b0613f7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:36 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:47070
I20260929 10:23:36.566504 26135 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8c9b520f97455a93fb943b0613f7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:36 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:60084
I20260929 10:23:36.566905 26241 tablet_replica.cc:330] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: stopping tablet replica
I20260929 10:23:36.566975 26242 tablet_replica.cc:330] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: stopping tablet replica
I20260929 10:23:36.566964 25867 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8c9b520f97455a93fb943b0613f7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:36 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:60238
I20260929 10:23:36.567269 26242 raft_consensus.cc:2229] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:36.567281 26241 raft_consensus.cc:2229] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:36.567574 26242 raft_consensus.cc:2258] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:36.567646 26243 tablet_replica.cc:330] T 8a8c9b520f97455a93fb943b0613f7b1 P f642f32582fb4ae18e15ce38391997f4: stopping tablet replica
I20260929 10:23:36.568038 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25804
I20260929 10:23:36.569422 26241 raft_consensus.cc:2258] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:36.570935 26241 ts_tablet_manager.cc:1905] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:36.573347 26241 ts_tablet_manager.cc:1918] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:36.573436 26241 log.cc:1196] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/8a8c9b520f97455a93fb943b0613f7b1
I20260929 10:23:36.573761 26241 ts_tablet_manager.cc:1939] T 8a8c9b520f97455a93fb943b0613f7b1 P 3955a338231e4479a336cc12ffa54eff: Deleting consensus metadata
I20260929 10:23:36.574878 25725 catalog_manager.cc:4841] TS 3955a338231e4479a336cc12ffa54eff (127.21.33.130:44325): tablet 8a8c9b520f97455a93fb943b0613f7b1 (table test-table [id=b3cb7413110c42ad8fa3f32d1b643677]) successfully deleted
W20260929 10:23:36.575199 25727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:37751 (error 108)
I20260929 10:23:36.576203 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.575927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37751 (local address 127.0.0.1:60254)
0929 10:23:36.576063 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:36.576132 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:37751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 10:23:36.576332 25727 catalog_manager.cc:4584] TS f642f32582fb4ae18e15ce38391997f4 (127.21.33.129:37751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a8c9b520f97455a93fb943b0613f7b1: Network error: Client connection negotiation failed: client connection to 127.21.33.129:37751: connect: Connection refused (error 111)
I20260929 10:23:36.576721 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25939
I20260929 10:23:36.579308 26242 ts_tablet_manager.cc:1905] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:36.583058 26242 ts_tablet_manager.cc:1918] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:36.583184 26242 log.cc:1196] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/8a8c9b520f97455a93fb943b0613f7b1
I20260929 10:23:36.583571 26242 ts_tablet_manager.cc:1939] T 8a8c9b520f97455a93fb943b0613f7b1 P 9e004dbb22b74eec82d9356732b0cc79: Deleting consensus metadata
I20260929 10:23:36.585006 25728 catalog_manager.cc:4841] TS 9e004dbb22b74eec82d9356732b0cc79 (127.21.33.131:40047): tablet 8a8c9b520f97455a93fb943b0613f7b1 (table test-table [id=b3cb7413110c42ad8fa3f32d1b643677]) successfully deleted
I20260929 10:23:36.585276 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26073
I20260929 10:23:36.590714 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 25710
2026-09-29T10:23:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2380 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T10:23:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:36Z Disabled control of system clock
I20260929 10:23:36.621814 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:44639
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:44639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:23:36.729813 26248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:36.730078 26248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:36.730116 26248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:36.734107 26248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:36.734170 26248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:36.734185 26248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:36.734210 26248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:36.734225 26248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:36.738983 26248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:44639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:36.740025 26248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:36.741442 26248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:23:36.749805 26248 server_base.cc:1034] running on GCE node
W20260929 10:23:36.749794 26254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:36.749759 26253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:36.749778 26256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:36.750643 26248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:36.751715 26248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:36.752887 26248 hybrid_clock.cc:648] HybridClock initialized: now 1790677416752868 us; error 33 us; skew 500 ppm
I20260929 10:23:36.758498 26248 webserver.cc:460] Webserver started at http://127.21.33.190:44783/ using document root <none> and password file <none>
I20260929 10:23:36.759248 26248 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:36.759318 26248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:36.759545 26248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:36.761716 26248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "751fe291adf5493382a39bbb1a3c578c"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "0abae1e48b73b3ddd1e7ced9fb77bac7"
server_key_iv: "64e3a2a3349ddaffcd13ac08a8b30240"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.762306 26248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "751fe291adf5493382a39bbb1a3c578c"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "0abae1e48b73b3ddd1e7ced9fb77bac7"
server_key_iv: "64e3a2a3349ddaffcd13ac08a8b30240"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.766031 26248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:36.768463 26262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.769825 26248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:36.769948 26248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "751fe291adf5493382a39bbb1a3c578c"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "0abae1e48b73b3ddd1e7ced9fb77bac7"
server_key_iv: "64e3a2a3349ddaffcd13ac08a8b30240"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.770061 26248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:36.774595 26248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:36.775178 26248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:36.775362 26248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:36.782541 26248 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:44639
I20260929 10:23:36.782647 26315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:44639 every 8 connection(s)
I20260929 10:23:36.783505 26248 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:36.786746 26316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:36.787281 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26248
I20260929 10:23:36.787410 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:36.787784 21638 external_mini_cluster.cc:1428] Setting key 2090cbcea15999f7fbcde4f3d15d90ed
I20260929 10:23:36.793412 26316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c: Bootstrap starting.
I20260929 10:23:36.793453 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:36.789279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.21.33.190:44639)
0929 10:23:36.789927 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:36.789941 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:36.789975 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:23:36.790623 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:36.790640 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:36.790986 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:36.791332 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:36.792147 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:36.792151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:36.792164 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:36.792174 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:36.792229 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:36.792253 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:36.792398 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:36.792469 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 10:23:36.792714 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":137,"threads_started":1}
I20260929 10:23:36.795943 26316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:36.796689 26316 log.cc:824] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:36.798602 26316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c: No bootstrap required, opened a new log
I20260929 10:23:36.802282 26316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751fe291adf5493382a39bbb1a3c578c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44639 } }
I20260929 10:23:36.802531 26316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:36.802618 26316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751fe291adf5493382a39bbb1a3c578c, State: Initialized, Role: FOLLOWER
I20260929 10:23:36.803035 26316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751fe291adf5493382a39bbb1a3c578c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44639 } }
I20260929 10:23:36.803198 26316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:36.803287 26316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:36.803409 26316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:36.804339 26316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751fe291adf5493382a39bbb1a3c578c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44639 } }
I20260929 10:23:36.804682 26316 leader_election.cc:302] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751fe291adf5493382a39bbb1a3c578c; no voters:
I20260929 10:23:36.804989 26316 leader_election.cc:288] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:36.805131 26321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:36.805409 26321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [term 1 LEADER]: Becoming Leader. State: Replica: 751fe291adf5493382a39bbb1a3c578c, State: Running, Role: LEADER
I20260929 10:23:36.805723 26321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751fe291adf5493382a39bbb1a3c578c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44639 } }
I20260929 10:23:36.806320 26316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:36.806782 26321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751fe291adf5493382a39bbb1a3c578c. Latest consensus state: current_term: 1 leader_uuid: "751fe291adf5493382a39bbb1a3c578c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751fe291adf5493382a39bbb1a3c578c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44639 } } }
I20260929 10:23:36.806905 26321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:36.807175 26322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "751fe291adf5493382a39bbb1a3c578c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751fe291adf5493382a39bbb1a3c578c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44639 } } }
I20260929 10:23:36.807282 26322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:36.807375 26325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:36.810447 26325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:36.815837 26325 catalog_manager.cc:1347] Generated new cluster ID: bd84b472c620469abc3efb12435efe36
I20260929 10:23:36.815910 26325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:36.832837 26325 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:36.833182 26325 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:36.838940 26325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c: Generated new TSK 0
I20260929 10:23:36.839596 26325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:23:36.848071 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44639
--builtin_ntp_servers=127.21.33.148:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:23:36.959661 26340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:36.959981 26340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:36.960065 26340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:36.964223 26340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:36.964308 26340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:36.964401 26340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:36.969300 26340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:36.970438 26340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:36.971879 26340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:36.982163 26346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:36.982170 26345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:36.982609 26340 server_base.cc:1034] running on GCE node
W20260929 10:23:36.982163 26348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:36.983086 26340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:36.984012 26340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:36.985301 26340 hybrid_clock.cc:648] HybridClock initialized: now 1790677416985285 us; error 29 us; skew 500 ppm
I20260929 10:23:36.987278 26340 webserver.cc:460] Webserver started at http://127.21.33.129:39955/ using document root <none> and password file <none>
I20260929 10:23:36.987888 26340 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:36.987982 26340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:36.988217 26340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:36.990372 26340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "c7d31b920e0343149f17ed60ea740f97"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "30d8fe9f388a3026657ff6013c60e0a0"
server_key_iv: "e74d09a40cbc18036aec955878b82fbb"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.990947 26340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "c7d31b920e0343149f17ed60ea740f97"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "30d8fe9f388a3026657ff6013c60e0a0"
server_key_iv: "e74d09a40cbc18036aec955878b82fbb"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.994728 26340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:23:36.997347 26354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:36.998590 26340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:36.998728 26340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "c7d31b920e0343149f17ed60ea740f97"
format_stamp: "Formatted at 2026-09-29 10:23:36 on dist-test-slave-2rbl"
server_key: "30d8fe9f388a3026657ff6013c60e0a0"
server_key_iv: "e74d09a40cbc18036aec955878b82fbb"
server_key_version: "encryptionkey@0"
I20260929 10:23:36.998842 26340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:37.003464 26340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:37.004076 26340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:37.004264 26340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:37.004809 26340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:37.005812 26340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:37.005887 26340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.005962 26340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:37.006006 26340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.014909 26340 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:38153
I20260929 10:23:37.014969 26468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:38153 every 8 connection(s)
I20260929 10:23:37.015882 26340 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:37.022183 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.017850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:59787 (local address 127.21.33.190:44639)
0929 10:23:37.017984 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.017988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.019078 (+ 1090us) server_negotiation.cc:407] Connection header received
0929 10:23:37.020377 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.020381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.020423 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.020505 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.021698 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.021702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.021704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.021717 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.021739 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.021748 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.021847 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.022017 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.022059 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:23:37.023053 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.018250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:44639 (local address 127.21.33.129:59787)
0929 10:23:37.018942 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.018983 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.020159 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.020687 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.020695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.020785 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.021444 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.021446 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.021487 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.021558 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.021867 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.021876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.021881 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.022034 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.022225 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260929 10:23:37.024333 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26340
I20260929 10:23:37.024372 26469 heartbeater.cc:347] Connected to a master server at 127.21.33.190:44639
I20260929 10:23:37.024479 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:37.024664 26469 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:37.024821 21638 external_mini_cluster.cc:1428] Setting key 1af2d4b512a01a0c4f55dc2b164aca8a
I20260929 10:23:37.025149 26469 heartbeater.cc:511] Master 127.21.33.190:44639 requested a full tablet report, sending...
I20260929 10:23:37.026789 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44639
--builtin_ntp_servers=127.21.33.148:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 10:23:37.026798 26279 ts_manager.cc:194] Registered new tserver with Master: c7d31b920e0343149f17ed60ea740f97 (127.21.33.129:38153)
W20260929 10:23:37.137260 26472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:37.137567 26472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:37.137610 26472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:37.141623 26472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:37.141680 26472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:37.141742 26472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:37.146538 26472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:37.147902 26472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:37.149297 26472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:37.159021 26480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:37.159186 26477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:37.159034 26478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:37.159354 26472 server_base.cc:1034] running on GCE node
I20260929 10:23:37.159880 26472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:37.160627 26472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:37.161805 26472 hybrid_clock.cc:648] HybridClock initialized: now 1790677417161788 us; error 28 us; skew 500 ppm
I20260929 10:23:37.163689 26472 webserver.cc:460] Webserver started at http://127.21.33.130:35095/ using document root <none> and password file <none>
I20260929 10:23:37.164278 26472 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:37.164338 26472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:37.164582 26472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:37.167667 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "5664d611bf594960853a3b2a6a64b17c"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "2b8cdc1914f0ecbaa9d77cd008dac2f2"
server_key_iv: "61146ce1ef64ae33b55a34d1e32ef084"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.168449 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "5664d611bf594960853a3b2a6a64b17c"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "2b8cdc1914f0ecbaa9d77cd008dac2f2"
server_key_iv: "61146ce1ef64ae33b55a34d1e32ef084"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.173645 26472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 10:23:37.176580 26486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.177860 26472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:37.177990 26472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "5664d611bf594960853a3b2a6a64b17c"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "2b8cdc1914f0ecbaa9d77cd008dac2f2"
server_key_iv: "61146ce1ef64ae33b55a34d1e32ef084"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.178110 26472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:37.182572 26472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:37.183158 26472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:37.183331 26472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:37.183845 26472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:37.184715 26472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:37.184787 26472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.184856 26472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:37.184909 26472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.193854 26472 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:37731
I20260929 10:23:37.193882 26600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:37731 every 8 connection(s)
I20260929 10:23:37.194752 26472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:37.200454 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.196481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:47317 (local address 127.21.33.190:44639)
0929 10:23:37.196656 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.196660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.197712 (+ 1052us) server_negotiation.cc:407] Connection header received
0929 10:23:37.198964 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.198967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.199004 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.199081 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.200120 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.200122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.200124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.200133 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.200148 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.200155 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.200223 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.200314 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.200349 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:23:37.201349 26603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.196890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:44639 (local address 127.21.33.130:47317)
0929 10:23:37.197572 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.197611 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.198776 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.199231 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.199239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.199324 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.199891 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.199893 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.199931 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.199997 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.200210 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.200213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.200216 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.200331 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.200506 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":119,"threads_started":1}
I20260929 10:23:37.202399 26601 heartbeater.cc:347] Connected to a master server at 127.21.33.190:44639
I20260929 10:23:37.202617 26601 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:37.203018 26601 heartbeater.cc:511] Master 127.21.33.190:44639 requested a full tablet report, sending...
I20260929 10:23:37.203332 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26472
I20260929 10:23:37.203450 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:37.203795 21638 external_mini_cluster.cc:1428] Setting key 01a6f6333edac69083fd56fa22f0e8d8
I20260929 10:23:37.204087 26279 ts_manager.cc:194] Registered new tserver with Master: 5664d611bf594960853a3b2a6a64b17c (127.21.33.130:37731)
I20260929 10:23:37.205847 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44639
--builtin_ntp_servers=127.21.33.148:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 10:23:37.318288 26604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:37.318579 26604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:37.318684 26604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:37.322649 26604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:37.322721 26604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:37.322801 26604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:37.327457 26604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:37.328459 26604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:37.329861 26604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:37.339401 26612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:37.339562 26604 server_base.cc:1034] running on GCE node
W20260929 10:23:37.339413 26609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:37.339412 26610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:37.340142 26604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:37.340741 26604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:37.341894 26604 hybrid_clock.cc:648] HybridClock initialized: now 1790677417341877 us; error 27 us; skew 500 ppm
I20260929 10:23:37.343648 26604 webserver.cc:460] Webserver started at http://127.21.33.131:41223/ using document root <none> and password file <none>
I20260929 10:23:37.344221 26604 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:37.344285 26604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:37.344519 26604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:37.346583 26604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "0a54111613814313afe34b6c193864ad"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "df3b46d6160111bd9f741932141ccd50"
server_key_iv: "6b05f2ea3b472d215e95d31fe11e7d8b"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.347169 26604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "0a54111613814313afe34b6c193864ad"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "df3b46d6160111bd9f741932141ccd50"
server_key_iv: "6b05f2ea3b472d215e95d31fe11e7d8b"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.350945 26604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 10:23:37.353453 26618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.354684 26604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:37.354818 26604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "0a54111613814313afe34b6c193864ad"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "df3b46d6160111bd9f741932141ccd50"
server_key_iv: "6b05f2ea3b472d215e95d31fe11e7d8b"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.354948 26604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:37.359515 26604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:37.360052 26604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:37.360275 26604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:37.360824 26604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:37.361743 26604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:37.361791 26604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.361848 26604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:37.361864 26604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.370270 26604 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:39379
I20260929 10:23:37.370288 26732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:39379 every 8 connection(s)
I20260929 10:23:37.371158 26604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:37.372000 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26604
I20260929 10:23:37.372118 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:37.372408 21638 external_mini_cluster.cc:1428] Setting key f5116cfc3c2b3b97b55e33183e36e77a
I20260929 10:23:37.378276 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.374118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:45711 (local address 127.21.33.190:44639)
0929 10:23:37.374257 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.374262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.375493 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 10:23:37.376735 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.376737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.376772 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.376846 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.377909 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.377912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.377914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.377925 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.377945 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.377952 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.378026 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.378137 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.378174 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:23:37.379380 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.374679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:44639 (local address 127.21.33.131:45711)
0929 10:23:37.375359 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.375398 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.376558 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.376997 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.377004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.377092 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.377670 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.377672 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.377712 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.377780 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.378024 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.378030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.378034 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.378161 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.378394 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260929 10:23:37.380455 26733 heartbeater.cc:347] Connected to a master server at 127.21.33.190:44639
I20260929 10:23:37.380715 26733 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:37.381134 26733 heartbeater.cc:511] Master 127.21.33.190:44639 requested a full tablet report, sending...
I20260929 10:23:37.382242 26278 ts_manager.cc:194] Registered new tserver with Master: 0a54111613814313afe34b6c193864ad (127.21.33.131:39379)
I20260929 10:23:37.386364 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:23:37.389645 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.388165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.21.33.190:44639)
0929 10:23:37.388301 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.388304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.388511 (+ 207us) server_negotiation.cc:407] Connection header received
0929 10:23:37.388601 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.388604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.388633 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.388700 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.389364 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.389367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.389369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.389379 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.389396 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.389404 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.389472 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.389521 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.389551 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:23:37.392377 26278 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:37.395349 26278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:37.406931 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.403653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:37731 (local address 127.0.0.1:60054)
0929 10:23:37.404116 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.404164 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.404280 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.405266 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.405276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.405361 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.406006 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.406008 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.406028 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.406098 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.406536 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.406544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.406549 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.406663 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.406770 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":75,"threads_started":1}
I20260929 10:23:37.407589 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.404735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.21.33.129:38153)
0929 10:23:37.405029 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.405040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.405063 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:23:37.405116 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.405124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.405338 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.405462 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.406634 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.406641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.406653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.406666 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.406713 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.406749 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.407272 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.407341 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.407413 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
I20260929 10:23:37.407608 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.405069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.21.33.131:39379)
0929 10:23:37.405560 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.405574 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.405601 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:23:37.405655 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.405661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.405837 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.405977 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.406789 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.406796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.406806 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.406817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.406857 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.406885 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.406988 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.407364 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.407418 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":202,"threads_started":1}
I20260929 10:23:37.407092 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.404392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:38153 (local address 127.0.0.1:37414)
0929 10:23:37.404687 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.404701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.404787 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.405966 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.405969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.405985 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.406426 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.406429 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.406440 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.406468 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.406843 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.406846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.406850 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.406915 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.406949 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":91,"threads_started":1}
I20260929 10:23:37.407279 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.403764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:39379 (local address 127.0.0.1:48872)
0929 10:23:37.404359 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.404373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.404442 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.406222 (+ 1780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.406225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.406241 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.406601 (+ 360us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.406602 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.406610 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.406631 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.407036 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.407041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.407047 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.407119 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.407150 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":104,"threads_started":1}
I20260929 10:23:37.408036 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.403763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.21.33.130:37731)
0929 10:23:37.404710 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.404731 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.404757 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:23:37.404828 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.404834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.405058 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.405233 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.406254 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.406264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.406281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.406297 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.406351 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.406390 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.406491 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.407276 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.407362 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":130,"threads_started":1}
I20260929 10:23:37.410780 26534 tablet_service.cc:1467] Processing CreateTablet for tablet 6f47670c0d754b258873b84604cda12f (DEFAULT_TABLE table=test-table [id=b8359778159d411fb5f253bb85928325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:37.410792 26403 tablet_service.cc:1467] Processing CreateTablet for tablet 6f47670c0d754b258873b84604cda12f (DEFAULT_TABLE table=test-table [id=b8359778159d411fb5f253bb85928325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:37.410887 26666 tablet_service.cc:1467] Processing CreateTablet for tablet 6f47670c0d754b258873b84604cda12f (DEFAULT_TABLE table=test-table [id=b8359778159d411fb5f253bb85928325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:37.411880 26666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f47670c0d754b258873b84604cda12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:37.411880 26403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f47670c0d754b258873b84604cda12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:37.411896 26534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f47670c0d754b258873b84604cda12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:37.418078 26750 tablet_bootstrap.cc:492] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Bootstrap starting.
I20260929 10:23:37.419525 26751 tablet_bootstrap.cc:492] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97: Bootstrap starting.
I20260929 10:23:37.420449 26750 tablet_bootstrap.cc:654] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:37.421321 26750 log.cc:824] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:37.421550 26752 tablet_bootstrap.cc:492] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Bootstrap starting.
I20260929 10:23:37.421630 26751 tablet_bootstrap.cc:654] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:37.422446 26751 log.cc:824] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:37.423300 26750 tablet_bootstrap.cc:492] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: No bootstrap required, opened a new log
I20260929 10:23:37.423511 26750 ts_tablet_manager.cc:1397] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:23:37.423645 26752 tablet_bootstrap.cc:654] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:37.424455 26752 log.cc:824] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:37.424566 26751 tablet_bootstrap.cc:492] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97: No bootstrap required, opened a new log
I20260929 10:23:37.424782 26751 ts_tablet_manager.cc:1397] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:23:37.426673 26752 tablet_bootstrap.cc:492] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: No bootstrap required, opened a new log
I20260929 10:23:37.426882 26752 ts_tablet_manager.cc:1397] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:37.427734 26750 raft_consensus.cc:348] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.427997 26750 raft_consensus.cc:374] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:37.428071 26750 raft_consensus.cc:729] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5664d611bf594960853a3b2a6a64b17c, State: Initialized, Role: FOLLOWER
I20260929 10:23:37.428543 26750 consensus_queue.cc:260] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.429265 26601 heartbeater.cc:503] Master 127.21.33.190:44639 was elected leader, sending a full tablet report...
I20260929 10:23:37.429366 26751 raft_consensus.cc:348] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.429625 26751 raft_consensus.cc:374] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:37.429642 26750 ts_tablet_manager.cc:1428] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:23:37.429685 26751 raft_consensus.cc:729] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d31b920e0343149f17ed60ea740f97, State: Initialized, Role: FOLLOWER
I20260929 10:23:37.430248 26751 consensus_queue.cc:260] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.431237 26751 ts_tablet_manager.cc:1428] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:23:37.431401 26752 raft_consensus.cc:348] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.431622 26469 heartbeater.cc:503] Master 127.21.33.190:44639 was elected leader, sending a full tablet report...
I20260929 10:23:37.431684 26752 raft_consensus.cc:374] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:37.431749 26752 raft_consensus.cc:729] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a54111613814313afe34b6c193864ad, State: Initialized, Role: FOLLOWER
I20260929 10:23:37.432168 26752 consensus_queue.cc:260] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.433157 26733 heartbeater.cc:503] Master 127.21.33.190:44639 was elected leader, sending a full tablet report...
I20260929 10:23:37.433178 26752 ts_tablet_manager.cc:1428] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 10:23:37.445640 26602 tablet.cc:2369] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:37.467761 26758 raft_consensus.cc:482] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:37.467990 26758 raft_consensus.cc:504] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.469120 26758 leader_election.cc:288] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5664d611bf594960853a3b2a6a64b17c (127.21.33.130:37731), c7d31b920e0343149f17ed60ea740f97 (127.21.33.129:38153)
I20260929 10:23:37.470943 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.469562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:51643 (local address 127.21.33.130:37731)
0929 10:23:37.469682 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.469688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.469706 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:23:37.469795 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.469799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.469839 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.469932 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.470568 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.470572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.470575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.470587 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.470612 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.470620 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.470714 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.470789 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.470826 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:23:37.470943 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.469447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:37731 (local address 127.21.33.131:51643)
0929 10:23:37.469595 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.469613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.469702 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.469959 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.469962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.469974 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.470395 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.470396 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.470406 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.470428 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.470699 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.470702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.470705 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.470781 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.470818 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 10:23:37.471621 26555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f47670c0d754b258873b84604cda12f" candidate_uuid: "0a54111613814313afe34b6c193864ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5664d611bf594960853a3b2a6a64b17c" is_pre_election: true
I20260929 10:23:37.471745 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.469585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:38613 (local address 127.21.33.129:38153)
0929 10:23:37.469709 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.469713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.470090 (+ 377us) server_negotiation.cc:407] Connection header received
0929 10:23:37.470408 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.470411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.470446 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.470545 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.471363 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.471368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.471376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.471388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.471405 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.471413 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.471508 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.471583 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.471621 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:23:37.471773 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.469447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:38153 (local address 127.21.33.131:38613)
0929 10:23:37.469983 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.469996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.470085 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.470571 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.470574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.470590 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:37.471187 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:23:37.471189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:37.471200 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:23:37.471228 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:37.471505 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:37.471508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:37.471511 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:23:37.471586 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:23:37.471621 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":164,"thread_start_us":132,"threads_started":1}
I20260929 10:23:37.471941 26555 raft_consensus.cc:2454] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a54111613814313afe34b6c193864ad in term 0.
I20260929 10:23:37.472393 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f47670c0d754b258873b84604cda12f" candidate_uuid: "0a54111613814313afe34b6c193864ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d31b920e0343149f17ed60ea740f97" is_pre_election: true
I20260929 10:23:37.472414 26619 leader_election.cc:302] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a54111613814313afe34b6c193864ad, 5664d611bf594960853a3b2a6a64b17c; no voters:
I20260929 10:23:37.472663 26423 raft_consensus.cc:2454] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a54111613814313afe34b6c193864ad in term 0.
I20260929 10:23:37.472687 26758 raft_consensus.cc:2781] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:37.472769 26758 raft_consensus.cc:482] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:37.472846 26758 raft_consensus.cc:3037] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:37.473996 26758 raft_consensus.cc:504] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.474390 26758 leader_election.cc:288] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [CANDIDATE]: Term 1 election: Requested vote from peers 5664d611bf594960853a3b2a6a64b17c (127.21.33.130:37731), c7d31b920e0343149f17ed60ea740f97 (127.21.33.129:38153)
I20260929 10:23:37.474725 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f47670c0d754b258873b84604cda12f" candidate_uuid: "0a54111613814313afe34b6c193864ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d31b920e0343149f17ed60ea740f97"
I20260929 10:23:37.474753 26555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f47670c0d754b258873b84604cda12f" candidate_uuid: "0a54111613814313afe34b6c193864ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5664d611bf594960853a3b2a6a64b17c"
I20260929 10:23:37.474850 26423 raft_consensus.cc:3037] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:37.474881 26555 raft_consensus.cc:3037] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:37.476114 26423 raft_consensus.cc:2454] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a54111613814313afe34b6c193864ad in term 1.
I20260929 10:23:37.476111 26555 raft_consensus.cc:2454] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a54111613814313afe34b6c193864ad in term 1.
I20260929 10:23:37.476385 26619 leader_election.cc:302] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a54111613814313afe34b6c193864ad, 5664d611bf594960853a3b2a6a64b17c; no voters:
I20260929 10:23:37.476580 26758 raft_consensus.cc:2781] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:37.476816 26758 raft_consensus.cc:686] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 1 LEADER]: Becoming Leader. State: Replica: 0a54111613814313afe34b6c193864ad, State: Running, Role: LEADER
I20260929 10:23:37.477182 26758 consensus_queue.cc:237] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } }
I20260929 10:23:37.480358 26277 catalog_manager.cc:5495] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a54111613814313afe34b6c193864ad (127.21.33.131). New cstate: current_term: 1 leader_uuid: "0a54111613814313afe34b6c193864ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a54111613814313afe34b6c193864ad" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:37.506108 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.504510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.21.33.131:39379)
0929 10:23:37.504636 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.504640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.504681 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:23:37.504838 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.504840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.504898 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.504969 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.505688 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.505691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.505693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.505703 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.505722 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.505730 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.505842 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.505954 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.505994 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:23:37.512157 26423 raft_consensus.cc:1261] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97 [term 1 FOLLOWER]: Refusing update from remote peer 0a54111613814313afe34b6c193864ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:37.512190 26555 raft_consensus.cc:1261] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 1 FOLLOWER]: Refusing update from remote peer 0a54111613814313afe34b6c193864ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:37.512920 26760 consensus_queue.cc:1035] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7d31b920e0343149f17ed60ea740f97" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:37.513130 26758 consensus_queue.cc:1035] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5664d611bf594960853a3b2a6a64b17c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:23:37.517093 26470 tablet.cc:2369] T 6f47670c0d754b258873b84604cda12f P c7d31b920e0343149f17ed60ea740f97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:37.521741 26764 mvcc.cc:205] Tried to move back new op lower bound from 7334614702122835968 to 7334614701989195776. Current Snapshot: MvccSnapshot[applied={T|T < 7334614702122835968}]
I20260929 10:23:37.524924 26763 mvcc.cc:205] Tried to move back new op lower bound from 7334614702122835968 to 7334614701989195776. Current Snapshot: MvccSnapshot[applied={T|T < 7334614702122835968 or (T in {7334614702122835968})}]
I20260929 10:23:37.529124 26278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:37.529333 26278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:37.531683 26278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 751fe291adf5493382a39bbb1a3c578c: Sending DeleteTablet for 3 replicas of tablet 6f47670c0d754b258873b84604cda12f
I20260929 10:23:37.532469 26534 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f47670c0d754b258873b84604cda12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:37 UTC) from {username='slave'} at 127.0.0.1:60054
I20260929 10:23:37.532605 26666 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f47670c0d754b258873b84604cda12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:37 UTC) from {username='slave'} at 127.0.0.1:48872
I20260929 10:23:37.532953 26403 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f47670c0d754b258873b84604cda12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:37 UTC) from {username='slave'} at 127.0.0.1:37414
I20260929 10:23:37.533116 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26340
I20260929 10:23:37.533753 26773 tablet_replica.cc:330] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: stopping tablet replica
I20260929 10:23:37.534058 26773 raft_consensus.cc:2229] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:37.534396 26773 raft_consensus.cc:2258] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:37.535857 26773 ts_tablet_manager.cc:1905] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:37.539355 26773 ts_tablet_manager.cc:1918] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:37.539517 26773 log.cc:1196] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/6f47670c0d754b258873b84604cda12f
I20260929 10:23:37.539947 26773 ts_tablet_manager.cc:1939] T 6f47670c0d754b258873b84604cda12f P 5664d611bf594960853a3b2a6a64b17c: Deleting consensus metadata
I20260929 10:23:37.541318 26263 catalog_manager.cc:4841] TS 5664d611bf594960853a3b2a6a64b17c (127.21.33.130:37731): tablet 6f47670c0d754b258873b84604cda12f (table test-table [id=b8359778159d411fb5f253bb85928325]) successfully deleted
I20260929 10:23:37.541471 26774 tablet_replica.cc:330] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: stopping tablet replica
W20260929 10:23:37.541778 26264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:38153 (error 108)
I20260929 10:23:37.541806 26774 raft_consensus.cc:2229] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:37.542143 26774 raft_consensus.cc:2258] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:37.542682 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26472
I20260929 10:23:37.542908 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.542535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:38153 (local address 127.0.0.1:37426)
0929 10:23:37.542732 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.542810 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:38153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 10:23:37.544027 26774 ts_tablet_manager.cc:1905] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:23:37.544866 26264 catalog_manager.cc:4584] TS c7d31b920e0343149f17ed60ea740f97 (127.21.33.129:38153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f47670c0d754b258873b84604cda12f: Network error: Client connection negotiation failed: client connection to 127.21.33.129:38153: connect: Connection refused (error 111)
I20260929 10:23:37.546627 26774 ts_tablet_manager.cc:1918] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:37.546722 26774 log.cc:1196] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/6f47670c0d754b258873b84604cda12f
I20260929 10:23:37.547036 26774 ts_tablet_manager.cc:1939] T 6f47670c0d754b258873b84604cda12f P 0a54111613814313afe34b6c193864ad: Deleting consensus metadata
I20260929 10:23:37.548177 26264 catalog_manager.cc:4841] TS 0a54111613814313afe34b6c193864ad (127.21.33.131:39379): tablet 6f47670c0d754b258873b84604cda12f (table test-table [id=b8359778159d411fb5f253bb85928325]) successfully deleted
I20260929 10:23:37.549592 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26604
I20260929 10:23:37.555258 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26248
2026-09-29T10:23:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (963 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 10:23:37.576678 21638 mini_oidc.cc:152] Starting JWKS server
I20260929 10:23:37.578370 21638 webserver.cc:460] Webserver started at https://127.0.0.1:39867/ using document root <none> and password file <none>
I20260929 10:23:37.578558 21638 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:23:37.578833 21638 webserver.cc:460] Webserver started at http://127.0.0.1:34781/ using document root <none> and password file <none>
2026-09-29T10:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:37Z Disabled control of system clock
I20260929 10:23:37.587822 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:34927
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:34927
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39867/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/testchainca.cert with env {}
W20260929 10:23:37.698376 26782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:37.698621 26782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:37.698668 26782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:37.703167 26782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:37.703223 26782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:37.703239 26782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:37.703256 26782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:37.703271 26782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:23:37.703286 26782 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:23:37.703300 26782 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39867/jwks.json
I20260929 10:23:37.708613 26782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:34927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:34927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39867/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:37.709601 26782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:37.710523 26782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:37.719271 26788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:37.719271 26787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:37.719269 26790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:37.719312 26782 server_base.cc:1034] running on GCE node
I20260929 10:23:37.720090 26782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:37.721247 26782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:37.722435 26782 hybrid_clock.cc:648] HybridClock initialized: now 1790677417722421 us; error 35 us; skew 500 ppm
I20260929 10:23:37.724260 26782 webserver.cc:460] Webserver started at http://127.21.33.190:45251/ using document root <none> and password file <none>
I20260929 10:23:37.724833 26782 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:37.724925 26782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:37.725149 26782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:37.727195 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "86dd888e24aa4ff59274f80e05fe3482"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "f4c7b09cab9834ca2f0677bc247088aa"
server_key_iv: "a82a7ad115ab86072643fe19c357066c"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.727751 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "86dd888e24aa4ff59274f80e05fe3482"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "f4c7b09cab9834ca2f0677bc247088aa"
server_key_iv: "a82a7ad115ab86072643fe19c357066c"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.731484 26782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:37.733888 26796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:37.735188 26782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:37.735366 26782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "86dd888e24aa4ff59274f80e05fe3482"
format_stamp: "Formatted at 2026-09-29 10:23:37 on dist-test-slave-2rbl"
server_key: "f4c7b09cab9834ca2f0677bc247088aa"
server_key_iv: "a82a7ad115ab86072643fe19c357066c"
server_key_version: "encryptionkey@0"
I20260929 10:23:37.735495 26782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:37.747036 26782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:37.747752 26782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:37.747962 26782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:37.755250 26782 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:34927
I20260929 10:23:37.755260 26848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:34927 every 8 connection(s)
I20260929 10:23:37.756274 26782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:37.759473 26849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:37.763449 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26782
I20260929 10:23:37.763691 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:37.763975 21638 external_mini_cluster.cc:1428] Setting key deed9ab681b21ee0052c5d960e5aa280
I20260929 10:23:37.765378 26849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482: Bootstrap starting.
I20260929 10:23:37.767544 26849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:37.768252 26849 log.cc:824] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:37.770037 26849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482: No bootstrap required, opened a new log
I20260929 10:23:37.773351 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.765300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.21.33.190:34927)
0929 10:23:37.765927 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.765940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.765974 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:23:37.766601 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.766616 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.766998 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.767344 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:37.767823 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:37.768839 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:37.769811 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:37.770333 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:37.770506 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:37.771354 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:37.771359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:37.771371 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:37.771924 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:37.771997 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:37.772019 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:37.772132 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:37.772284 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 10:23:37.772568 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260929 10:23:37.774078 26849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86dd888e24aa4ff59274f80e05fe3482" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34927 } }
I20260929 10:23:37.774354 26849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:37.774406 26849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86dd888e24aa4ff59274f80e05fe3482, State: Initialized, Role: FOLLOWER
I20260929 10:23:37.774803 26849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86dd888e24aa4ff59274f80e05fe3482" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34927 } }
I20260929 10:23:37.774946 26849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:37.775009 26849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:37.775125 26849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:37.776162 26849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86dd888e24aa4ff59274f80e05fe3482" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34927 } }
I20260929 10:23:37.776469 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:23:37.776489 26849 leader_election.cc:302] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86dd888e24aa4ff59274f80e05fe3482; no voters:
I20260929 10:23:37.776836 26849 leader_election.cc:288] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:37.776906 26854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:37.777089 26854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [term 1 LEADER]: Becoming Leader. State: Replica: 86dd888e24aa4ff59274f80e05fe3482, State: Running, Role: LEADER
I20260929 10:23:37.777401 26854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86dd888e24aa4ff59274f80e05fe3482" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34927 } }
I20260929 10:23:37.778096 26849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:37.778541 26854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86dd888e24aa4ff59274f80e05fe3482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86dd888e24aa4ff59274f80e05fe3482" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34927 } } }
I20260929 10:23:37.778659 26854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:37.778851 26854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86dd888e24aa4ff59274f80e05fe3482. Latest consensus state: current_term: 1 leader_uuid: "86dd888e24aa4ff59274f80e05fe3482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86dd888e24aa4ff59274f80e05fe3482" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34927 } } }
I20260929 10:23:37.778950 26854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:37.779093 26858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:37.782303 26858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:37.787684 26858 catalog_manager.cc:1347] Generated new cluster ID: 2adf881e21094a6f9e65831966031bab
I20260929 10:23:37.787766 26858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:37.813411 26858 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:37.814316 26858 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:37.828145 26858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86dd888e24aa4ff59274f80e05fe3482: Generated new TSK 0
I20260929 10:23:37.828896 26858 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 10:23:37.864516 26879 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 10:23:37.864636 26879 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 10:23:37.865075 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:37.840780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.21.33.190:34927)
0929 10:23:37.840933 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:37.840938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:37.841159 (+ 221us) server_negotiation.cc:407] Connection header received
0929 10:23:37.841410 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:37.841413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:37.841461 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:37.841549 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:23:37.841851 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:37.842680 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:37.843873 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:37.844091 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:37.844166 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:37.844521 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:23:37.864677 (+ 20156us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 10:23:37.864964 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 10:23:37.865183 26852 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58698: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 10:23:37.866500 26878 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:37.840696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34927 (local address 127.0.0.1:58698)
0929 10:23:37.841059 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:37.841079 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:37.841242 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:37.841589 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:37.841592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:37.841608 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:23:37.841742 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:37.841749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:37.842795 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:37.842799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:37.843762 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:37.843772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:37.844201 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:37.844205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:37.844295 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:37.844345 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:23:37.864915 (+ 20570us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 10:23:37.865175 (+ 260us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.33.190:34927: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":126,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.33.190:34927: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 10:23:37.867267 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26782
2026-09-29T10:23:37Z chronyd exiting
I20260929 10:23:37.904177 21638 test_util.cc:175] -----------------------------------------------
I20260929 10:23:37.904268 21638 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790677387835684-21638-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 10:23:37.905332 21638 mini_oidc.cc:152] Starting JWKS server
I20260929 10:23:37.906162 21638 webserver.cc:460] Webserver started at https://127.0.0.1:45651/ using document root <none> and password file <none>
I20260929 10:23:37.906243 21638 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:23:37.906473 21638 webserver.cc:460] Webserver started at http://127.0.0.1:46287/ using document root <none> and password file <none>
2026-09-29T10:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:37Z Disabled control of system clock
I20260929 10:23:37.915712 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:45815
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:42605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:45815
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45651/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/testchainca.cert with env {}
W20260929 10:23:38.025452 26885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:38.025727 26885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:38.025767 26885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:38.029722 26885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:38.029783 26885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:38.029799 26885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:38.029816 26885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:38.029831 26885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:23:38.029847 26885 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:23:38.029860 26885 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45651/jwks.json
I20260929 10:23:38.035187 26885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:45815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:45815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45651/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:38.036196 26885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:38.037231 26885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:38.045823 26890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:38.045895 26885 server_base.cc:1034] running on GCE node
W20260929 10:23:38.045921 26891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:38.045823 26893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:38.046689 26885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:38.047809 26885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:38.048971 26885 hybrid_clock.cc:648] HybridClock initialized: now 1790677418048957 us; error 26 us; skew 500 ppm
I20260929 10:23:38.050897 26885 webserver.cc:460] Webserver started at http://127.21.33.190:45361/ using document root <none> and password file <none>
I20260929 10:23:38.051487 26885 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:38.051551 26885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:38.051771 26885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:38.053896 26885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "788625cd36f347cbb3503a8283766694"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "1b92bf728a58dde6828da63055ead928"
server_key_iv: "512dbee1a939a0be785fe2b96e9920ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.054469 26885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "788625cd36f347cbb3503a8283766694"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "1b92bf728a58dde6828da63055ead928"
server_key_iv: "512dbee1a939a0be785fe2b96e9920ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.058172 26885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:38.060653 26899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:38.062171 26885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:23:38.062304 26885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "788625cd36f347cbb3503a8283766694"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "1b92bf728a58dde6828da63055ead928"
server_key_iv: "512dbee1a939a0be785fe2b96e9920ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.062431 26885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:38.076714 26885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:38.077525 26885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:38.077720 26885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:38.084446 26885 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:45815
I20260929 10:23:38.084456 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:45815 every 8 connection(s)
I20260929 10:23:38.085412 26885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:38.088577 26952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:38.091437 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26885
I20260929 10:23:38.091572 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:38.091892 21638 external_mini_cluster.cc:1428] Setting key 31b89558a072f7cca8a78c1a7fc0f302
I20260929 10:23:38.094779 26952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694: Bootstrap starting.
I20260929 10:23:38.096937 26952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:38.097731 26952 log.cc:824] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:38.099798 26952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694: No bootstrap required, opened a new log
I20260929 10:23:38.101392 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:38.093286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.21.33.190:45815)
0929 10:23:38.093918 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:38.093935 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:38.093971 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:23:38.094579 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:38.094595 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:38.095044 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:38.095385 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:38.095806 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.096776 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.097796 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.098498 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.098678 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.099585 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:38.099598 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:38.099610 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:38.100050 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:38.100105 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:38.100125 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:38.100233 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:38.100375 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 10:23:38.100625 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260929 10:23:38.103924 26952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788625cd36f347cbb3503a8283766694" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45815 } }
I20260929 10:23:38.104182 26952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:38.104243 26952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788625cd36f347cbb3503a8283766694, State: Initialized, Role: FOLLOWER
I20260929 10:23:38.104355 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:23:38.104739 26952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788625cd36f347cbb3503a8283766694" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45815 } }
I20260929 10:23:38.104892 26952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:38.104979 26952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:38.105075 26952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:38.106102 26952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788625cd36f347cbb3503a8283766694" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45815 } }
I20260929 10:23:38.106446 26952 leader_election.cc:302] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788625cd36f347cbb3503a8283766694; no voters:
I20260929 10:23:38.106767 26952 leader_election.cc:288] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:38.106909 26957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:38.107162 26957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [term 1 LEADER]: Becoming Leader. State: Replica: 788625cd36f347cbb3503a8283766694, State: Running, Role: LEADER
I20260929 10:23:38.107438 26957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788625cd36f347cbb3503a8283766694" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45815 } }
I20260929 10:23:38.108006 26952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:38.109076 26959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788625cd36f347cbb3503a8283766694. Latest consensus state: current_term: 1 leader_uuid: "788625cd36f347cbb3503a8283766694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788625cd36f347cbb3503a8283766694" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45815 } } }
I20260929 10:23:38.109236 26959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:38.109634 26966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:38.109549 26958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788625cd36f347cbb3503a8283766694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788625cd36f347cbb3503a8283766694" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45815 } } }
I20260929 10:23:38.109700 26958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:38.112690 26966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:38.118121 26966 catalog_manager.cc:1347] Generated new cluster ID: 4e25d09baa014bd2bfda4400c45f0fe6
I20260929 10:23:38.118197 26966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:38.127997 26966 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:38.128855 26966 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:38.133104 26966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788625cd36f347cbb3503a8283766694: Generated new TSK 0
I20260929 10:23:38.133771 26966 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 10:23:38.161080 26982 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 10:23:38.161175 26982 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 10:23:38.161944 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:38.146400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.21.33.190:45815)
0929 10:23:38.146534 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:38.146540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:38.146711 (+ 171us) server_negotiation.cc:407] Connection header received
0929 10:23:38.146929 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:38.146932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:38.146979 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:38.147092 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:23:38.147348 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.147886 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.148761 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.148995 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.149108 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.149459 (+ 351us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:23:38.161498 (+ 12039us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 10:23:38.161785 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50220: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 10:23:38.162011 26955 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50220: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 10:23:38.162021 26981 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:38.146298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:45815 (local address 127.0.0.1:50220)
0929 10:23:38.146625 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:38.146637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:38.146790 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:38.147105 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:38.147109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:38.147118 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:23:38.147239 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:38.147246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.148007 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.148010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:38.148636 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:38.148643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.149124 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.149128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:38.149248 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.149275 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:23:38.161686 (+ 12411us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 10:23:38.161852 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.33.190:45815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":119,"threads_started":1}
I20260929 10:23:38.162848 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26885
2026-09-29T10:23:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 10:23:38.202010 21638 mini_oidc.cc:152] Starting JWKS server
I20260929 10:23:38.202857 21638 webserver.cc:460] Webserver started at https://127.0.0.1:39455/ using document root <none> and password file <none>
I20260929 10:23:38.202934 21638 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 10:23:38.203179 21638 webserver.cc:460] Webserver started at http://127.0.0.1:41857/ using document root <none> and password file <none>
2026-09-29T10:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:38Z Disabled control of system clock
I20260929 10:23:38.211647 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:32991
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:44385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:32991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39455/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 10:23:38.319525 26988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:38.319839 26988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:38.319913 26988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:38.323916 26988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:38.324015 26988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:38.324044 26988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:38.324083 26988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:38.324118 26988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:23:38.324149 26988 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:23:38.324179 26988 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39455/jwks.json
I20260929 10:23:38.328877 26988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:32991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39455/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:38.330235 26988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:38.331702 26988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:38.340166 26993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:38.340183 26988 server_base.cc:1034] running on GCE node
W20260929 10:23:38.340124 26994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:38.340137 26996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:38.340904 26988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:38.342033 26988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:38.343202 26988 hybrid_clock.cc:648] HybridClock initialized: now 1790677418343189 us; error 25 us; skew 500 ppm
I20260929 10:23:38.345130 26988 webserver.cc:460] Webserver started at http://127.21.33.190:37923/ using document root <none> and password file <none>
I20260929 10:23:38.345726 26988 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:38.345790 26988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:38.345947 26988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:38.347884 26988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "4899ba7bd6e64d53841ccc627a5c5a39"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "d3e0f2b3f6ccee6190b32fa1e44add71"
server_key_iv: "64d1226a55d506a4b80d8c95efbbf5fb"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.348399 26988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "4899ba7bd6e64d53841ccc627a5c5a39"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "d3e0f2b3f6ccee6190b32fa1e44add71"
server_key_iv: "64d1226a55d506a4b80d8c95efbbf5fb"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.352010 26988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:23:38.354382 27002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:38.355588 26988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:38.355731 26988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "4899ba7bd6e64d53841ccc627a5c5a39"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "d3e0f2b3f6ccee6190b32fa1e44add71"
server_key_iv: "64d1226a55d506a4b80d8c95efbbf5fb"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.355844 26988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:38.373155 26988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:38.373966 26988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:38.374178 26988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:38.381261 26988 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:32991
I20260929 10:23:38.381282 27054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:32991 every 8 connection(s)
I20260929 10:23:38.382247 26988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:38.385443 27055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:38.387436 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 26988
I20260929 10:23:38.387583 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:38.387900 21638 external_mini_cluster.cc:1428] Setting key f9cad899dce6c44bba99058bce60f75b
I20260929 10:23:38.391407 27055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39: Bootstrap starting.
I20260929 10:23:38.393728 27055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:38.394465 27055 log.cc:824] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:38.396447 27055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39: No bootstrap required, opened a new log
I20260929 10:23:38.397064 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:38.389273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.21.33.190:32991)
0929 10:23:38.389940 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:38.389956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:38.389989 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:23:38.390625 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:38.390642 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:38.391045 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:38.391418 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:38.391854 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.392815 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.393778 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.394300 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.394473 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.395343 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:38.395348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:38.395362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:38.395750 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:38.395808 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:38.395828 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:38.395940 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:38.396055 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 10:23:38.396318 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":138,"threads_started":1}
I20260929 10:23:38.400054 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:23:38.400629 27055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 32991 } }
I20260929 10:23:38.400870 27055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:38.400943 27055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4899ba7bd6e64d53841ccc627a5c5a39, State: Initialized, Role: FOLLOWER
I20260929 10:23:38.401432 27055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 32991 } }
I20260929 10:23:38.401598 27055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:38.401669 27055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:38.401774 27055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:38.402712 27055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 32991 } }
I20260929 10:23:38.403052 27055 leader_election.cc:302] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4899ba7bd6e64d53841ccc627a5c5a39; no voters:
I20260929 10:23:38.403383 27055 leader_election.cc:288] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:38.403496 27060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:38.403766 27060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [term 1 LEADER]: Becoming Leader. State: Replica: 4899ba7bd6e64d53841ccc627a5c5a39, State: Running, Role: LEADER
I20260929 10:23:38.404078 27060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 32991 } }
I20260929 10:23:38.404806 27055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:38.405992 27061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 32991 } } }
I20260929 10:23:38.406139 27061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:38.406417 27062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4899ba7bd6e64d53841ccc627a5c5a39. Latest consensus state: current_term: 1 leader_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4899ba7bd6e64d53841ccc627a5c5a39" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 32991 } } }
I20260929 10:23:38.406497 27062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:38.406509 27069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:38.409360 27069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:38.414736 27069 catalog_manager.cc:1347] Generated new cluster ID: 1394a45413c043bd968bc4a620a64ad2
I20260929 10:23:38.414822 27069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:38.438524 27069 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:38.439502 27069 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:38.446386 27069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4899ba7bd6e64d53841ccc627a5c5a39: Generated new TSK 0
I20260929 10:23:38.447122 27069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:23:38.464843 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:38.459384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.21.33.190:32991)
0929 10:23:38.459550 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:38.459556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:38.459799 (+ 243us) server_negotiation.cc:407] Connection header received
0929 10:23:38.459962 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:38.459964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:38.460015 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:38.460108 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:23:38.460344 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.461002 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.462285 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.462563 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.462747 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.463065 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:23:38.464461 (+ 1396us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 10:23:38.464699 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 10:23:38.464912 27058 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 10:23:38.466715 27084 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:38.459331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:32991 (local address 127.0.0.1:41540)
0929 10:23:38.459687 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:38.459703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:38.459832 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:38.460129 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:38.460135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:38.460145 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:23:38.460255 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:38.460262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.461120 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.461124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:38.462149 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:38.462159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.462776 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.462780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:38.462872 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.462900 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:23:38.464625 (+ 1725us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 10:23:38.466539 (+ 1914us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.33.190:32991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39455/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":1715,"thread_start_us":137,"threads_started":1}
I20260929 10:23:38.467648 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 26988
E20260929 10:23:38.476475 27085 webserver.cc:533] Webserver: sslize failed:
E20260929 10:23:38.476540 27085 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T10:23:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T10:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:38Z Disabled control of system clock
I20260929 10:23:38.516053 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39537
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:39537
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/testchainca.cert with env {}
W20260929 10:23:38.625285 27089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:38.625586 27089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:38.625656 27089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:38.629746 27089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:38.629830 27089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:38.629856 27089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:38.629905 27089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:38.629961 27089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 10:23:38.629997 27089 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 10:23:38.630028 27089 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 10:23:38.635453 27089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:39537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:38.636523 27089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:38.637598 27089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:38.646216 27094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:38.646219 27095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:38.646260 27097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:38.646473 27089 server_base.cc:1034] running on GCE node
I20260929 10:23:38.647267 27089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:38.648322 27089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:38.649488 27089 hybrid_clock.cc:648] HybridClock initialized: now 1790677418649467 us; error 34 us; skew 500 ppm
I20260929 10:23:38.651329 27089 webserver.cc:460] Webserver started at http://127.21.33.190:39153/ using document root <none> and password file <none>
I20260929 10:23:38.651865 27089 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:38.651925 27089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:38.652078 27089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:38.654152 27089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "a72c5e38af844c7c8ea6d00f53d5d34b"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "7ae4b16e55a80816287b80598bc68617"
server_key_iv: "e83d5d0a29c2b30855e95b7ddb7bd1e0"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.654657 27089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "a72c5e38af844c7c8ea6d00f53d5d34b"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "7ae4b16e55a80816287b80598bc68617"
server_key_iv: "e83d5d0a29c2b30855e95b7ddb7bd1e0"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.658217 27089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:23:38.660571 27103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:38.662263 27089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:23:38.662395 27089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "a72c5e38af844c7c8ea6d00f53d5d34b"
format_stamp: "Formatted at 2026-09-29 10:23:38 on dist-test-slave-2rbl"
server_key: "7ae4b16e55a80816287b80598bc68617"
server_key_iv: "e83d5d0a29c2b30855e95b7ddb7bd1e0"
server_key_version: "encryptionkey@0"
I20260929 10:23:38.662513 27089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:38.684755 27089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:38.685446 27089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:38.685601 27089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:38.692442 27089 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:39537
I20260929 10:23:38.692464 27155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:39537 every 8 connection(s)
I20260929 10:23:38.693423 27089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:38.696448 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:38.701607 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27089
I20260929 10:23:38.701751 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:38.702033 21638 external_mini_cluster.cc:1428] Setting key 50ce9b447f82223c0251aa73a1ecac3d
I20260929 10:23:38.702188 27156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b: Bootstrap starting.
I20260929 10:23:38.704834 27156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:38.705690 27156 log.cc:824] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:38.707695 27156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b: No bootstrap required, opened a new log
I20260929 10:23:38.711609 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:38.703388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.21.33.190:39537)
0929 10:23:38.704066 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:38.704080 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:38.704119 (+ 39us) server_negotiation.cc:407] Connection header received
0929 10:23:38.704772 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:38.704790 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:38.705321 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:38.705662 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:38.706091 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.707081 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.708218 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.708704 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.708875 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.709820 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:38.709831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:38.709844 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:23:38.710255 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:38.710316 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:23:38.710335 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:38.710443 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:38.710591 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 10:23:38.710866 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":143,"threads_started":1}
I20260929 10:23:38.711781 27156 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39537 } }
I20260929 10:23:38.712080 27156 raft_consensus.cc:374] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:38.712164 27156 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72c5e38af844c7c8ea6d00f53d5d34b, State: Initialized, Role: FOLLOWER
I20260929 10:23:38.712678 27156 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39537 } }
I20260929 10:23:38.712849 27156 raft_consensus.cc:388] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:38.712944 27156 raft_consensus.cc:482] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:38.713056 27156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:38.714287 27156 raft_consensus.cc:504] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39537 } }
I20260929 10:23:38.714720 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:23:38.714681 27156 leader_election.cc:302] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a72c5e38af844c7c8ea6d00f53d5d34b; no voters:
I20260929 10:23:38.715065 27156 leader_election.cc:288] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:38.715190 27161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:38.715378 27161 raft_consensus.cc:686] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [term 1 LEADER]: Becoming Leader. State: Replica: a72c5e38af844c7c8ea6d00f53d5d34b, State: Running, Role: LEADER
I20260929 10:23:38.715862 27161 consensus_queue.cc:237] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39537 } }
I20260929 10:23:38.716439 27156 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:38.717123 27161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72c5e38af844c7c8ea6d00f53d5d34b. Latest consensus state: current_term: 1 leader_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39537 } } }
I20260929 10:23:38.717293 27161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:38.717816 27162 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72c5e38af844c7c8ea6d00f53d5d34b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39537 } } }
I20260929 10:23:38.717947 27162 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:38.719290 27168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:38.722116 27168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:38.727339 27168 catalog_manager.cc:1347] Generated new cluster ID: ff086c0646244f51ace03a10a90c783a
I20260929 10:23:38.727412 27168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:38.737306 27168 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:38.738534 27168 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:38.745021 27168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a72c5e38af844c7c8ea6d00f53d5d34b: Generated new TSK 0
I20260929 10:23:38.745853 27168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:23:39.219326 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:38.754883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.21.33.190:39537)
0929 10:23:38.755018 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:38.755023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:38.755172 (+ 149us) server_negotiation.cc:407] Connection header received
0929 10:23:38.755332 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:38.755336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:38.755394 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:38.755517 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0929 10:23:38.755721 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.756536 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.757501 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.757857 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.757956 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.758242 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 10:23:39.218929 (+460687us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 10:23:39.219195 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51642: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 10:23:39.219399 27159 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51642: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 10:23:39.219583 27185 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:38.754769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39537 (local address 127.0.0.1:51642)
0929 10:23:38.755087 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:38.755100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:38.755202 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:38.755520 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:38.755522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:38.755528 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0929 10:23:38.755621 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:38.755625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.756679 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.756682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:38.757386 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:38.757396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:38.757985 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:38.757988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:38.758048 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:38.758070 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 10:23:39.219173 (+461103us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 10:23:39.219343 (+ 170us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.33.190:39537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":95,"threads_started":1}
I20260929 10:23:39.220141 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27089
2026-09-29T10:23:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (731 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (131 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:39 dist-test-slave-2rbl krb5kdc[27197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:39 dist-test-slave-2rbl krb5kdc[27197](info): set up 2 sockets
Sep 29 10:23:39 dist-test-slave-2rbl krb5kdc[27197](info): commencing operation
krb5kdc: starting...
W20260929 10:23:40.518172 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:40 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:40.644758 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35995
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:40.752899 27210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:40.753160 27210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:40.753263 27210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:40.757165 27210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:40.757237 27210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:40.757257 27210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:40.757274 27210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:40.757289 27210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:40.762451 27210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:40.763414 27210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:40.764344 27210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:40.772361 27218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:40.772557 27215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:40.772557 27216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:40.772626 27210 server_base.cc:1034] running on GCE node
I20260929 10:23:40.773375 27210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:40.774410 27210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:40.775591 27210 hybrid_clock.cc:648] HybridClock initialized: now 1790677420775570 us; error 33 us; skew 500 ppm
Sep 29 10:23:40 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677420, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:40.778383 27210 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:40.779506 27210 webserver.cc:460] Webserver started at http://127.21.33.190:38765/ using document root <none> and password file <none>
I20260929 10:23:40.780197 27210 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:40.780283 27210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:40.780494 27210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:40.782579 27210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "423caa0c6dfa465b91dfd87c87f45df1"
format_stamp: "Formatted at 2026-09-29 10:23:40 on dist-test-slave-2rbl"
server_key: "935217a540e1d3ae17ac447c2bcac4e9"
server_key_iv: "f24bf9d8787cc7250513d2b98e5d879d"
server_key_version: "encryptionkey@0"
I20260929 10:23:40.783151 27210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "423caa0c6dfa465b91dfd87c87f45df1"
format_stamp: "Formatted at 2026-09-29 10:23:40 on dist-test-slave-2rbl"
server_key: "935217a540e1d3ae17ac447c2bcac4e9"
server_key_iv: "f24bf9d8787cc7250513d2b98e5d879d"
server_key_version: "encryptionkey@0"
I20260929 10:23:40.786868 27210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:40.789376 27225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:40.790728 27210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:40.790910 27210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "423caa0c6dfa465b91dfd87c87f45df1"
format_stamp: "Formatted at 2026-09-29 10:23:40 on dist-test-slave-2rbl"
server_key: "935217a540e1d3ae17ac447c2bcac4e9"
server_key_iv: "f24bf9d8787cc7250513d2b98e5d879d"
server_key_version: "encryptionkey@0"
I20260929 10:23:40.791029 27210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:40.805965 27210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:40.809219 27210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:40.809419 27210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:40.816428 27210 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:35995
I20260929 10:23:40.816452 27277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:35995 every 8 connection(s)
I20260929 10:23:40.817353 27210 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:40.819916 27278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:40.820621 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27210
I20260929 10:23:40.820760 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:40.821027 21638 external_mini_cluster.cc:1428] Setting key b9783d8f6acbf9843d866e5601e0eec3
I20260929 10:23:40.826514 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1: Bootstrap starting.
I20260929 10:23:40.828950 27278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:40.829754 27278 log.cc:824] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:23:40 dist-test-slave-2rbl krb5kdc[27197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:40.831832 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1: No bootstrap required, opened a new log
I20260929 10:23:40.835342 27278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423caa0c6dfa465b91dfd87c87f45df1" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35995 } }
I20260929 10:23:40.835510 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:40.822435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.21.33.190:35995)
0929 10:23:40.822850 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:40.822868 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:40.822912 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:23:40.823724 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:40.823746 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:40.824136 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:40.824495 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:40.825143 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:40.826596 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:40.827426 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:40.828104 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:40.828268 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:40.830890 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:40.830895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:40.830908 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:40.830933 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:40.832679 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:40.833189 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:40.833243 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:40.833249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:40.833366 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:40.833651 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:40.833654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:40.833656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:40.834040 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:40.834251 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:40.834528 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 10:23:40.834805 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":98,"threads_started":1}
I20260929 10:23:40.835551 27278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:40.835637 27278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 423caa0c6dfa465b91dfd87c87f45df1, State: Initialized, Role: FOLLOWER
I20260929 10:23:40.836112 27278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423caa0c6dfa465b91dfd87c87f45df1" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35995 } }
I20260929 10:23:40.836234 27278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:40.836318 27278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:40.836457 27278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:40.837419 27278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423caa0c6dfa465b91dfd87c87f45df1" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35995 } }
I20260929 10:23:40.837762 27278 leader_election.cc:302] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 423caa0c6dfa465b91dfd87c87f45df1; no voters:
I20260929 10:23:40.838066 27278 leader_election.cc:288] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:40.838155 27283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:40.838385 27283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [term 1 LEADER]: Becoming Leader. State: Replica: 423caa0c6dfa465b91dfd87c87f45df1, State: Running, Role: LEADER
I20260929 10:23:40.838726 27283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423caa0c6dfa465b91dfd87c87f45df1" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35995 } }
I20260929 10:23:40.839314 27278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:40.840335 27284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "423caa0c6dfa465b91dfd87c87f45df1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423caa0c6dfa465b91dfd87c87f45df1" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35995 } } }
I20260929 10:23:40.840459 27284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:40.840785 27285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 423caa0c6dfa465b91dfd87c87f45df1. Latest consensus state: current_term: 1 leader_uuid: "423caa0c6dfa465b91dfd87c87f45df1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "423caa0c6dfa465b91dfd87c87f45df1" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35995 } } }
I20260929 10:23:40.840910 27285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:40.840950 27292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:40.844017 27292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:40.849637 27292 catalog_manager.cc:1347] Generated new cluster ID: ad6d28117ed54f49bd18eed125bdfa37
I20260929 10:23:40.849720 27292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:40.867483 27292 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:40.868803 27292 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:40.880909 27292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 423caa0c6dfa465b91dfd87c87f45df1: Generated new TSK 0
I20260929 10:23:40.881806 27292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:40.915970 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35995
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:41.027318 27304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:41.027565 27304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:41.027607 27304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:41.031636 27304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:41.031693 27304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:41.031757 27304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:41.036823 27304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:41.038159 27304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:41.039211 27304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:41.049095 27310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:41.049098 27309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:41.049300 27304 server_base.cc:1034] running on GCE node
W20260929 10:23:41.049057 27312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:41.049827 27304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:41.050438 27304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:41.051621 27304 hybrid_clock.cc:648] HybridClock initialized: now 1790677421051610 us; error 33 us; skew 500 ppm
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.054319 27304 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:41.055097 27304 webserver.cc:460] Webserver started at http://127.21.33.129:34955/ using document root <none> and password file <none>
I20260929 10:23:41.055603 27304 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:41.055660 27304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:41.055820 27304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:41.057819 27304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "a35e67920ca94557a2c960a43ccee4ce"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "d4ff9d3e11c1bf8b167cdacb98b17fd9"
server_key_iv: "a621ba29636a60902b5b4dbfe0b7c388"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.058329 27304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "a35e67920ca94557a2c960a43ccee4ce"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "d4ff9d3e11c1bf8b167cdacb98b17fd9"
server_key_iv: "a621ba29636a60902b5b4dbfe0b7c388"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.061946 27304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:23:41.064270 27319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.065531 27304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 10:23:41.065658 27304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "a35e67920ca94557a2c960a43ccee4ce"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "d4ff9d3e11c1bf8b167cdacb98b17fd9"
server_key_iv: "a621ba29636a60902b5b4dbfe0b7c388"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.065758 27304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:41.074792 27304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:41.077557 27304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:41.077756 27304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:41.078339 27304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:41.079111 27304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:41.079156 27304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.079229 27304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:41.079260 27304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.088686 27304 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:46359
I20260929 10:23:41.088712 27432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:46359 every 8 connection(s)
I20260929 10:23:41.089634 27304 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:41.092101 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27304
I20260929 10:23:41.092240 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:41.092581 21638 external_mini_cluster.cc:1428] Setting key fed5b7143beb95a13c56f0e1b29b55f3
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:23:41.104321 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.091533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:53829 (local address 127.21.33.190:35995)
0929 10:23:41.091675 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.091679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.092425 (+ 746us) server_negotiation.cc:407] Connection header received
0929 10:23:41.093966 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.093970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.094022 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.094117 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.095253 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.095989 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.097146 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.097626 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.097751 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.100489 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:41.100493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:41.100495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:41.100523 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:41.102258 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.102829 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.102836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.102841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.102939 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.103247 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.103250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.103251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.103419 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:41.103493 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:41.104072 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 10:23:41.104184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:23:41.105336 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.091787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35995 (local address 127.21.33.129:53829)
0929 10:23:41.092256 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:41.092310 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:41.093755 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:41.094280 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:41.094287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:41.094632 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:41.095096 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:41.095108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.096129 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.096131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:41.097027 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:41.097037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.097788 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.097790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:41.097937 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.098572 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:41.098589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:41.100308 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:41.102387 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:41.102392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:41.102405 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:41.102728 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:41.103027 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:41.103029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:41.103030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:41.103162 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:41.103506 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:41.103511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:41.103841 (+ 330us) client_negotiation.cc:769] Sending connection context
0929 10:23:41.104025 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 10:23:41.104291 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":138,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:23:41.106600 27433 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35995
I20260929 10:23:41.106886 27433 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:41.107422 27433 heartbeater.cc:511] Master 127.21.33.190:35995 requested a full tablet report, sending...
I20260929 10:23:41.109061 27242 ts_manager.cc:194] Registered new tserver with Master: a35e67920ca94557a2c960a43ccee4ce (127.21.33.129:46359)
I20260929 10:23:41.110788 27242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:53829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:41.124776 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35995
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:41.235936 27438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:41.236207 27438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:41.236251 27438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:41.240549 27438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:41.240617 27438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:41.240729 27438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:41.246200 27438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:41.247260 27438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:41.248287 27438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:41.258459 27444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:41.258474 27446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:41.258472 27443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:41.258536 27438 server_base.cc:1034] running on GCE node
I20260929 10:23:41.259156 27438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:41.259838 27438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:41.261023 27438 hybrid_clock.cc:648] HybridClock initialized: now 1790677421260982 us; error 55 us; skew 500 ppm
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.263847 27438 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:41.264753 27438 webserver.cc:460] Webserver started at http://127.21.33.130:44569/ using document root <none> and password file <none>
I20260929 10:23:41.265352 27438 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:41.265419 27438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:41.265586 27438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:41.267567 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "45b3b6e797fa4bb8a38a7f564268531b"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "91999536dc2f8b2edd95cc5d50c256ac"
server_key_iv: "96bbc96d987c75573f14af2b51a06a7f"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.268066 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "45b3b6e797fa4bb8a38a7f564268531b"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "91999536dc2f8b2edd95cc5d50c256ac"
server_key_iv: "96bbc96d987c75573f14af2b51a06a7f"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.271930 27438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 10:23:41.274253 27453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.275429 27438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:41.275548 27438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "45b3b6e797fa4bb8a38a7f564268531b"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "91999536dc2f8b2edd95cc5d50c256ac"
server_key_iv: "96bbc96d987c75573f14af2b51a06a7f"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.275650 27438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:41.302471 27438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:41.305470 27438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:41.305670 27438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:41.306290 27438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:41.307104 27438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:41.307149 27438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.307216 27438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:41.307250 27438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.316120 27438 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:43055
I20260929 10:23:41.316150 27566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:43055 every 8 connection(s)
I20260929 10:23:41.317003 27438 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:41.321164 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27438
I20260929 10:23:41.321306 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:41.321645 21638 external_mini_cluster.cc:1428] Setting key bbb3bf1cf605a104f7bfe6777ae87c86
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:41.331308 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.318920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:57599 (local address 127.21.33.190:35995)
0929 10:23:41.319056 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.319060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.319881 (+ 821us) server_negotiation.cc:407] Connection header received
0929 10:23:41.321187 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.321227 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.321297 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.321390 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.322830 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.323564 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.324694 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.324930 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.325045 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.327454 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:41.327461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:41.327468 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:41.327501 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:41.329231 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.329709 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.329713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.329716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.329815 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.330138 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.330143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.330146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.330395 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:41.330504 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:41.330967 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 10:23:41.331106 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:41.332067 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.319299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35995 (local address 127.21.33.130:57599)
0929 10:23:41.319724 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:41.319767 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:41.320962 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:41.321722 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:41.321733 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:41.322197 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:41.322671 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:41.322685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.323696 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.323699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:41.324563 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:41.324573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.325084 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.325088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:41.325317 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.325880 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:41.325897 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:41.327286 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:41.329349 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:41.329353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:41.329365 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:41.329606 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:41.329914 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:41.329916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:41.329918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:41.330042 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:41.330500 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:41.330505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:41.330731 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 10:23:41.330880 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 10:23:41.331121 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260929 10:23:41.333164 27567 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35995
I20260929 10:23:41.333426 27567 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:41.333966 27567 heartbeater.cc:511] Master 127.21.33.190:35995 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:23:41.335208 27242 ts_manager.cc:194] Registered new tserver with Master: 45b3b6e797fa4bb8a38a7f564268531b (127.21.33.130:43055)
I20260929 10:23:41.336012 27242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:57599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:41.351692 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35995
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:41.460913 27572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:41.461153 27572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:41.461213 27572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:41.465247 27572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:41.465305 27572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:41.465404 27572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:41.470846 27572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:41.471868 27572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:41.472882 27572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:41.482307 27577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:41.482486 27572 server_base.cc:1034] running on GCE node
W20260929 10:23:41.482317 27578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:41.482808 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:41.483176 27572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:41.483724 27572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:41.484887 27572 hybrid_clock.cc:648] HybridClock initialized: now 1790677421484875 us; error 26 us; skew 500 ppm
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.487568 27572 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:41.488382 27572 webserver.cc:460] Webserver started at http://127.21.33.131:32943/ using document root <none> and password file <none>
I20260929 10:23:41.488925 27572 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:41.488986 27572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:41.489295 27572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:41.491298 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "5d6e36ed28c447a59cc711797db2355e"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "8e6fd3e0e35a3d9e5585c56e1fc85780"
server_key_iv: "10a2ff7a2ac7e83db119782ba1b2a380"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.491868 27572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "5d6e36ed28c447a59cc711797db2355e"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "8e6fd3e0e35a3d9e5585c56e1fc85780"
server_key_iv: "10a2ff7a2ac7e83db119782ba1b2a380"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.495676 27572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:23:41.498119 27587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.499285 27572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:41.499416 27572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "5d6e36ed28c447a59cc711797db2355e"
format_stamp: "Formatted at 2026-09-29 10:23:41 on dist-test-slave-2rbl"
server_key: "8e6fd3e0e35a3d9e5585c56e1fc85780"
server_key_iv: "10a2ff7a2ac7e83db119782ba1b2a380"
server_key_version: "encryptionkey@0"
I20260929 10:23:41.499517 27572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:41.527392 27572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:41.530347 27572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:41.530560 27572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:41.531174 27572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:41.532008 27572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:41.532054 27572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.532094 27572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:41.532151 27572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:41.541059 27572 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:43265
I20260929 10:23:41.541142 27700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:43265 every 8 connection(s)
I20260929 10:23:41.542047 27572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:41.547860 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27572
I20260929 10:23:41.547983 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:41.548269 21638 external_mini_cluster.cc:1428] Setting key a445f9cac97017b47fafef4435e27daa
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:41.555460 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.543840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:56715 (local address 127.21.33.190:35995)
0929 10:23:41.543979 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.543983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.544794 (+ 811us) server_negotiation.cc:407] Connection header received
0929 10:23:41.546147 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.546150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.546217 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.546323 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.547428 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.548044 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.549560 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.549788 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.549898 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.552231 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:41.552234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:41.552236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:41.552258 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:41.553512 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.553997 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.554001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.554003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.554081 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.554418 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.554422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.554424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.554593 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:41.554701 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:41.555177 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 10:23:41.555298 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:23:41.556205 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.544207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35995 (local address 127.21.33.131:56715)
0929 10:23:41.544640 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:41.544680 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:41.545926 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:41.546490 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:41.546497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:41.546838 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:41.547290 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:41.547302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.548169 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.548172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:41.549461 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:41.549475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.549929 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.549933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:41.550110 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.550685 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:41.550706 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:41.552060 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:41.553622 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:41.553628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:41.553641 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:41.553893 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:41.554180 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:41.554182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:41.554184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:41.554321 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:41.554697 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:41.554722 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:41.554946 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 10:23:41.555123 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 10:23:41.555421 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260929 10:23:41.557219 27701 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35995
I20260929 10:23:41.557421 27701 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:41.557825 27701 heartbeater.cc:511] Master 127.21.33.190:35995 requested a full tablet report, sending...
I20260929 10:23:41.558843 27242 ts_manager.cc:194] Registered new tserver with Master: 5d6e36ed28c447a59cc711797db2355e (127.21.33.131:43265)
I20260929 10:23:41.559403 27242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:56715
I20260929 10:23:41.562101 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/test-admin.keytab.
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.578833 21638 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 10:23:41.581583 27708 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:41.580135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.21.33.129:46359)
0929 10:23:41.580472 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.580486 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.580509 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:23:41.580556 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.580561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.580739 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.580855 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.581398 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39754: BlockingRecv error: recv got EOF from 127.0.0.1:39754 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
W20260929 10:23:41.581614 21638 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc' of type FILE
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.582890 21638 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.583784 21638 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:41.592681 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.584598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.21.33.129:46359)
0929 10:23:41.584716 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.584720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.584736 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:41.584856 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.584858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.584894 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.584957 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.585503 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.586283 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.587061 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.587420 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.587497 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.589736 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:41.589745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:41.589758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:41.589780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:41.591234 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.591605 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.591610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.591611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.591682 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.591898 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.591900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.591901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.592114 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:41.592267 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:41.592428 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 10:23:41.592560 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 10:23:41.596288 27708 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:41.595470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.21.33.129:46359)
0929 10:23:41.595605 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.595608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.595669 (+ 61us) server_negotiation.cc:407] Connection header received
0929 10:23:41.595752 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.595754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.595788 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.595847 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.596177 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39772: BlockingRecv error: recv got EOF from 127.0.0.1:39772 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 10:23:41.596387 21638 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc' of type FILE
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.597404 21638 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.598253 21638 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:41.606483 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.599032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.21.33.129:46359)
0929 10:23:41.599145 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.599149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.599261 (+ 112us) server_negotiation.cc:407] Connection header received
0929 10:23:41.599359 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.599363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.599403 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.599483 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.600022 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.600524 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.601368 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.601596 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.601704 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.603858 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:41.603861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:41.603863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:41.603882 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:41.605124 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.605527 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.605529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.605530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.605600 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.605836 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.605838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.605839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.605981 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:41.606118 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:41.606289 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 10:23:41.606377 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 10:23:41.608928 27708 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:41.608020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.21.33.129:46359)
0929 10:23:41.608143 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.608147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.608197 (+ 50us) server_negotiation.cc:407] Connection header received
0929 10:23:41.608328 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.608331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.608371 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.608464 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.608813 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39788: BlockingRecv error: recv got EOF from 127.0.0.1:39788 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 10:23:41.609270 21638 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790677387835684-21638-0/krb5kdc/krb5cc)
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.610208 21638 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:41.611037 21638 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:23:41.618749 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:41.611802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.21.33.129:46359)
0929 10:23:41.611922 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:41.611925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:41.611938 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:41.612035 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:41.612037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:41.612068 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:41.612122 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:41.612526 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.612998 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.613794 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:41.613992 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:41.614062 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:41.615976 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:41.615979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:41.615980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:41.615999 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:41.617402 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.617745 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.617747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.617748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.617814 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:41.618025 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:41.618027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:41.618028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:41.618165 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:41.618255 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:41.618458 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 10:23:41.618568 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:23:41.619699 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27304
I20260929 10:23:41.626075 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27438
I20260929 10:23:41.631866 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27572
I20260929 10:23:41.636852 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27210
2026-09-29T10:23:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2164 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27721](info): set up 2 sockets
Sep 29 10:23:41 dist-test-slave-2rbl krb5kdc[27721](info): commencing operation
krb5kdc: starting...
W20260929 10:23:42.678189 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:42 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab.
Entry for principal oryx/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab.
I20260929 10:23:42.800709 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35965
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab
--principal=oryx/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:42.909443 27734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:42.909754 27734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:42.909849 27734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:42.913897 27734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:42.913975 27734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:42.914012 27734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:42.914070 27734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:42.914111 27734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:42.919307 27734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab
--principal=oryx/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:42.920361 27734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:42.921437 27734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:42.929884 27739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:42.929955 27734 server_base.cc:1034] running on GCE node
W20260929 10:23:42.929888 27740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:42.929878 27742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:42.930671 27734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:42.931793 27734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:42.932982 27734 hybrid_clock.cc:648] HybridClock initialized: now 1790677422932967 us; error 28 us; skew 500 ppm
Sep 29 10:23:42 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677422, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:42.938971 27734 init.cc:377] Logged in from keytab as oryx/127.21.33.190@KRBTEST.COM (short username oryx)
I20260929 10:23:42.940192 27734 webserver.cc:460] Webserver started at http://127.21.33.190:41285/ using document root <none> and password file <none>
I20260929 10:23:42.940923 27734 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:42.940994 27734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:42.941162 27734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:42.943257 27734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "d47f8e27fada4f2d8fb7199955d5eb1c"
format_stamp: "Formatted at 2026-09-29 10:23:42 on dist-test-slave-2rbl"
server_key: "2a0db48e9f004bbc859ff359430484b6"
server_key_iv: "57eac26cc2dd6172df813fa785d9adcb"
server_key_version: "encryptionkey@0"
I20260929 10:23:42.943778 27734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "d47f8e27fada4f2d8fb7199955d5eb1c"
format_stamp: "Formatted at 2026-09-29 10:23:42 on dist-test-slave-2rbl"
server_key: "2a0db48e9f004bbc859ff359430484b6"
server_key_iv: "57eac26cc2dd6172df813fa785d9adcb"
server_key_version: "encryptionkey@0"
I20260929 10:23:42.947635 27734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:23:42.950697 27749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:42.951898 27734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:42.952003 27734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "d47f8e27fada4f2d8fb7199955d5eb1c"
format_stamp: "Formatted at 2026-09-29 10:23:42 on dist-test-slave-2rbl"
server_key: "2a0db48e9f004bbc859ff359430484b6"
server_key_iv: "57eac26cc2dd6172df813fa785d9adcb"
server_key_version: "encryptionkey@0"
I20260929 10:23:42.952140 27734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:42.964718 27734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:42.967851 27734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:42.968050 27734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:42.975191 27734 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:35965
I20260929 10:23:42.975473 27801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:35965 every 8 connection(s)
I20260929 10:23:42.976192 27734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:42.977260 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27734
I20260929 10:23:42.977396 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:42.977702 21638 external_mini_cluster.cc:1428] Setting key 00279ea4b52a6196afb5d973692eae9c
I20260929 10:23:42.979574 27802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:42.985033 27802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: Bootstrap starting.
Sep 29 10:23:42 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:42.987377 27802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:42.988189 27802 log.cc:824] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:42.990424 27802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: No bootstrap required, opened a new log
I20260929 10:23:42.991447 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:42.979098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.21.33.190:35965)
0929 10:23:42.979541 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:42.979556 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:42.979590 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:23:42.980268 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:42.980285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:42.980600 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:42.980934 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:42.981760 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:42.982741 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:42.983582 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:42.984046 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:42.984216 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:42.986743 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:42.986749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:42.986761 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:42.986786 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:42.988638 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:42.989182 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:42.989187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:42.989191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:42.989348 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:42.989628 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:42.989630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:42.989631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:42.989999 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:42.990149 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:42.990436 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 10:23:42.990766 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":148,"threads_started":1}
I20260929 10:23:42.994204 27802 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } }
I20260929 10:23:42.994410 27802 raft_consensus.cc:374] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:42.994489 27802 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47f8e27fada4f2d8fb7199955d5eb1c, State: Initialized, Role: FOLLOWER
I20260929 10:23:42.994872 27802 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } }
I20260929 10:23:42.995019 27802 raft_consensus.cc:388] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:42.995081 27802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:42.995194 27802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:42.996063 27802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } }
I20260929 10:23:42.996400 27802 leader_election.cc:302] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d47f8e27fada4f2d8fb7199955d5eb1c; no voters:
I20260929 10:23:42.996688 27802 leader_election.cc:288] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:42.996886 27807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:42.997107 27807 raft_consensus.cc:686] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 1 LEADER]: Becoming Leader. State: Replica: d47f8e27fada4f2d8fb7199955d5eb1c, State: Running, Role: LEADER
I20260929 10:23:42.997439 27807 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } }
I20260929 10:23:42.997985 27802 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:42.999210 27809 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47f8e27fada4f2d8fb7199955d5eb1c. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } }
I20260929 10:23:42.999228 27808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } }
I20260929 10:23:42.999356 27809 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:42.999360 27808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:42.999792 27816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:43.002658 27816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:43.008074 27816 catalog_manager.cc:1347] Generated new cluster ID: 3d8c5231345e459d85e898a741d21cdf
I20260929 10:23:43.008149 27816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:43.015892 27816 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:43.016770 27816 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:43.028316 27816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: Generated new TSK 0
I20260929 10:23:43.029031 27816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab.
Entry for principal oryx/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab.
I20260929 10:23:43.060060 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35965
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab
--principal=oryx/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:43.172334 27828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:43.172719 27828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:43.172811 27828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:43.177382 27828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:43.177444 27828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:43.177541 27828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:43.182822 27828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab
--principal=oryx/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:43.183848 27828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:43.184854 27828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:43.194152 27836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:43.194103 27833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:43.194408 27828 server_base.cc:1034] running on GCE node
W20260929 10:23:43.194090 27834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:43.194785 27828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:43.195514 27828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:43.196671 27828 hybrid_clock.cc:648] HybridClock initialized: now 1790677423196653 us; error 29 us; skew 500 ppm
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:43.199541 27828 init.cc:377] Logged in from keytab as oryx/127.21.33.129@KRBTEST.COM (short username oryx)
I20260929 10:23:43.200387 27828 webserver.cc:460] Webserver started at http://127.21.33.129:34671/ using document root <none> and password file <none>
I20260929 10:23:43.200881 27828 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:43.200935 27828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:43.201088 27828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:43.203078 27828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "2cb6f3e06f364173bc2e0715ddec5754"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "9e377d8dfce20a15e1239f622dbdeb95"
server_key_iv: "90cb9c0049423afbec8059eff024389e"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.203589 27828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "2cb6f3e06f364173bc2e0715ddec5754"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "9e377d8dfce20a15e1239f622dbdeb95"
server_key_iv: "90cb9c0049423afbec8059eff024389e"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.207255 27828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:43.209486 27843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.210672 27828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:23:43.210816 27828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "2cb6f3e06f364173bc2e0715ddec5754"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "9e377d8dfce20a15e1239f622dbdeb95"
server_key_iv: "90cb9c0049423afbec8059eff024389e"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.210923 27828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:43.228026 27828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:43.231557 27828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:43.231789 27828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:43.232403 27828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:43.233290 27828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:43.233361 27828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.233436 27828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:43.233489 27828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.242556 27828 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:35209
I20260929 10:23:43.242632 27956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:35209 every 8 connection(s)
I20260929 10:23:43.243486 27828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:43.246343 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27828
I20260929 10:23:43.246467 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:43.246809 21638 external_mini_cluster.cc:1428] Setting key b41d57a7d6c8203fcb09b5480797c1bf
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:23:43.259161 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.245331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:49171 (local address 127.21.33.190:35965)
0929 10:23:43.245469 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.245472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.246400 (+ 928us) server_negotiation.cc:407] Connection header received
0929 10:23:43.248044 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.248046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.248114 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.248196 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.249454 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.250222 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.252020 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.252240 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.252317 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.255210 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.255213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.255215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.255240 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.257011 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.257591 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.257594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.257596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.257684 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.258081 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.258083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.258084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.258271 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.258363 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.258925 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.259036 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.130@KRBTEST.COM" created.
I20260929 10:23:43.260334 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.245716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:49171)
0929 10:23:43.246206 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.246267 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.247824 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.248393 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.248404 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.248772 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.249307 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.249323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.250359 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.250364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.251895 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.251909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.252402 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.252409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.252643 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.253299 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.253325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.255029 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.257132 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.257138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.257154 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.257479 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.257842 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.257844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.257846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.257986 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.258370 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.258375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.258651 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.258870 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.259133 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260929 10:23:43.262351 27957 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35965
I20260929 10:23:43.262598 27957 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:43.263093 27957 heartbeater.cc:511] Master 127.21.33.190:35965 requested a full tablet report, sending...
I20260929 10:23:43.264581 27766 ts_manager.cc:194] Registered new tserver with Master: 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:43.265805 27766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:49171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab.
Entry for principal oryx/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab.
I20260929 10:23:43.276271 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35965
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab
--principal=oryx/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:43.384541 27962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:43.384861 27962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:43.384940 27962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:43.388926 27962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:43.389026 27962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:43.389122 27962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:43.394804 27962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab
--principal=oryx/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:43.395983 27962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:43.397047 27962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:43.406703 27970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:43.406703 27967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:43.406783 27968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:43.406939 27962 server_base.cc:1034] running on GCE node
I20260929 10:23:43.407519 27962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:43.408165 27962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:43.409373 27962 hybrid_clock.cc:648] HybridClock initialized: now 1790677423409344 us; error 30 us; skew 500 ppm
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:43.412146 27962 init.cc:377] Logged in from keytab as oryx/127.21.33.130@KRBTEST.COM (short username oryx)
I20260929 10:23:43.413093 27962 webserver.cc:460] Webserver started at http://127.21.33.130:38753/ using document root <none> and password file <none>
I20260929 10:23:43.413734 27962 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:43.413956 27962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:43.414245 27962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:43.416450 27962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "0f177df46dbe4c1fbef83392d6da2f2d"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "27dc5b52fe5d46afbe9a882dc34da2de"
server_key_iv: "22de473718ea999cc21fb308514c9ac1"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.417035 27962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "0f177df46dbe4c1fbef83392d6da2f2d"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "27dc5b52fe5d46afbe9a882dc34da2de"
server_key_iv: "22de473718ea999cc21fb308514c9ac1"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.420915 27962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:23:43.423477 27977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.424795 27962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:43.424957 27962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "0f177df46dbe4c1fbef83392d6da2f2d"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "27dc5b52fe5d46afbe9a882dc34da2de"
server_key_iv: "22de473718ea999cc21fb308514c9ac1"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.425079 27962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:43.449427 27962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:43.452955 27962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:43.453187 27962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:43.453840 27962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:43.454650 27962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:43.454702 27962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.454768 27962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:43.454823 27962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.463667 27962 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:35193
I20260929 10:23:43.463723 28090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:35193 every 8 connection(s)
I20260929 10:23:43.464562 27962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:43.472399 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 27962
I20260929 10:23:43.472539 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:43.472911 21638 external_mini_cluster.cc:1428] Setting key 0df67178d4776c8594b0a207e96788f4
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:43.478946 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.466375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:54649 (local address 127.21.33.190:35965)
0929 10:23:43.466507 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.466517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.467298 (+ 781us) server_negotiation.cc:407] Connection header received
0929 10:23:43.468613 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.468616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.468662 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.468744 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.470098 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.470668 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.471841 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.472071 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.472185 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.475033 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.475040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.475046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.475078 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.476644 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.477306 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.477309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.477311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.477385 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.477784 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.477786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.477788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.477953 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.478026 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.478654 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.478810 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:23:43.480108 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.466681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:54649)
0929 10:23:43.467135 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.467190 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.468421 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.468940 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.468948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.469427 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.469939 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.469954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.470795 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.470798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.471732 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.471742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.472221 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.472224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.472392 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.473128 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.473146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.474869 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.476780 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.476787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.476803 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.477180 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.477479 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.477484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.477489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.477674 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.478053 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.478062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.478393 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.478607 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.478985 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
I20260929 10:23:43.481345 28091 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35965
I20260929 10:23:43.481592 28091 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:43.482069 28091 heartbeater.cc:511] Master 127.21.33.190:35965 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:23:43.483130 27766 ts_manager.cc:194] Registered new tserver with Master: 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193)
I20260929 10:23:43.483697 27766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:54649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab.
Entry for principal oryx/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab.
I20260929 10:23:43.501092 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35965
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab
--principal=oryx/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:43.612871 28096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:43.613142 28096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:43.613257 28096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:43.617558 28096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:43.617620 28096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:43.617723 28096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:43.622964 28096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab
--principal=oryx/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:43.623983 28096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:43.625015 28096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:43.634914 28101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:43.634924 28102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:43.635278 28104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:43.635722 28096 server_base.cc:1034] running on GCE node
I20260929 10:23:43.636193 28096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:43.636807 28096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:43.637974 28096 hybrid_clock.cc:648] HybridClock initialized: now 1790677423637961 us; error 28 us; skew 500 ppm
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:43.640537 28096 init.cc:377] Logged in from keytab as oryx/127.21.33.131@KRBTEST.COM (short username oryx)
I20260929 10:23:43.641386 28096 webserver.cc:460] Webserver started at http://127.21.33.131:40823/ using document root <none> and password file <none>
I20260929 10:23:43.641899 28096 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:43.641956 28096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:43.642110 28096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:43.644047 28096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "5ccdcd9e0d9a437e9678399ebd420b5e"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "7849226ad648b2a5900f55904974dc86"
server_key_iv: "f3804d5678fc027d14b812801ecf0217"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.644534 28096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "5ccdcd9e0d9a437e9678399ebd420b5e"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "7849226ad648b2a5900f55904974dc86"
server_key_iv: "f3804d5678fc027d14b812801ecf0217"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.648217 28096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:23:43.650537 28111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.651724 28096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:43.651824 28096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "5ccdcd9e0d9a437e9678399ebd420b5e"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "7849226ad648b2a5900f55904974dc86"
server_key_iv: "f3804d5678fc027d14b812801ecf0217"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.651904 28096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:43.663517 28096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:43.666246 28096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:43.666388 28096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:43.666872 28096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:43.667660 28096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:43.667706 28096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.667750 28096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:43.667773 28096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.676419 28096 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:46599
I20260929 10:23:43.676450 28224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:46599 every 8 connection(s)
I20260929 10:23:43.677289 28096 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:43.677531 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 28096
I20260929 10:23:43.677672 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:43.677960 21638 external_mini_cluster.cc:1428] Setting key 52630840fc62988fba257fba635ef6ac
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:43.691105 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.679137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:47681 (local address 127.21.33.190:35965)
0929 10:23:43.679291 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.679296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.680400 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 10:23:43.681811 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.681814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.681865 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.681942 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.683048 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.683572 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.684705 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.684933 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.685046 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.687625 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.687637 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.687639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.687661 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.688920 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.689500 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.689505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.689507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.689587 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.689925 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.689929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.689930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.690108 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.690218 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.690807 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.690941 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:23:43.692054 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.679670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:47681)
0929 10:23:43.680253 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.680318 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.681606 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.682112 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.682119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.682461 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.682908 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.682920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.683696 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.683700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.684578 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.684587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.685080 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.685084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.685314 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.685909 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.685924 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.687450 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.689035 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.689040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.689054 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.689399 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.689678 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.689682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.689684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.689823 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.690213 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.690219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.690500 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.690691 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.691020 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":138,"threads_started":1}
I20260929 10:23:43.693058 28225 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35965
I20260929 10:23:43.693351 28225 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:43.693792 28225 heartbeater.cc:511] Master 127.21.33.190:35965 requested a full tablet report, sending...
I20260929 10:23:43.694679 27766 ts_manager.cc:194] Registered new tserver with Master: 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131:46599)
I20260929 10:23:43.695190 27766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:47681
I20260929 10:23:43.703784 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.33.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.189.keytab.
Entry for principal oryx/127.21.33.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.189.keytab.
I20260929 10:23:43.732024 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:45771
--webserver_interface=127.21.33.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.189.keytab
--principal=oryx/127.21.33.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35965,127.21.33.189:45771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:43.841182 28230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:43.841511 28230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:43.841583 28230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:43.845574 28230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:43.845623 28230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:43.845642 28230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:43.845659 28230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:43.845695 28230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:43.850867 28230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35965,127.21.33.189:45771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.189.keytab
--principal=oryx/127.21.33.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:45771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.33.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
W20260929 10:23:43.851037 28230 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.33.190:35965,127.21.33.189:45771'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 10:23:43.851899 28230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:43.852906 28230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:43.861516 28236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:43.861557 28235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:43.861569 28238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:43.861750 28230 server_base.cc:1034] running on GCE node
I20260929 10:23:43.862286 28230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:43.863346 28230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:43.864504 28230 hybrid_clock.cc:648] HybridClock initialized: now 1790677423864488 us; error 30 us; skew 500 ppm
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:43.867235 28230 init.cc:377] Logged in from keytab as oryx/127.21.33.189@KRBTEST.COM (short username oryx)
I20260929 10:23:43.868100 28230 webserver.cc:460] Webserver started at http://127.21.33.189:37259/ using document root <none> and password file <none>
I20260929 10:23:43.868602 28230 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:43.868659 28230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:43.868830 28230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:43.870870 28230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/instance:
uuid: "0b76f60281ac4622af2631809c3d238c"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "86a48b810704bb11979849f2a71e04dd"
server_key_iv: "12179135079cd415fad44c09922f06ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.871373 28230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal/instance:
uuid: "0b76f60281ac4622af2631809c3d238c"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "86a48b810704bb11979849f2a71e04dd"
server_key_iv: "12179135079cd415fad44c09922f06ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.875094 28230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 10:23:43.877362 28245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:43.878583 28230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:23:43.878692 28230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
uuid: "0b76f60281ac4622af2631809c3d238c"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "86a48b810704bb11979849f2a71e04dd"
server_key_iv: "12179135079cd415fad44c09922f06ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:43.878782 28230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:43.900941 28230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:43.904424 28230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:43.904605 28230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:43.912184 28230 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.189:45771
I20260929 10:23:43.912191 28297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.189:45771 every 8 connection(s)
I20260929 10:23:43.913364 28230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
I20260929 10:23:43.916194 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:43.917917 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 28230
I20260929 10:23:43.920135 28298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.33.189@KRBTEST.COM
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.189@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:43.932154 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.922454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.21.33.190:35965)
0929 10:23:43.922579 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.922584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.922865 (+ 281us) server_negotiation.cc:407] Connection header received
0929 10:23:43.923188 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.923190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.923240 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.923328 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.924115 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.924784 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.925970 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.926179 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.926269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.928837 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.928841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.928843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.928868 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.930397 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.930875 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.930880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.930882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.930961 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.931245 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.931248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.931250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.931455 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.931566 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.931877 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.931991 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:23:43.932600 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.919453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.21.33.189:45771)
0929 10:23:43.919840 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.919856 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.919891 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:23:43.920499 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.920516 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.920836 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.921163 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.922083 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.923407 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.924450 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.925097 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.925269 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.928077 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.928087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.928099 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.928123 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.929814 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.930233 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.930243 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.930248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.930342 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.930623 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.930628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.930629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.931004 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.931219 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.931457 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.931813 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":134,"threads_started":1}
I20260929 10:23:43.932596 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.922392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55096)
0929 10:23:43.922749 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.922787 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.923060 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.923374 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.923379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.923843 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.923988 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.923995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.924912 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.924914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.925847 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.925858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.926339 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.926343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.926484 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.926999 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.927014 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.928640 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.930526 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.930535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.930540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.930771 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.931060 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.931063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.931065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.931149 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.931564 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.931571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.931680 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.931824 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.931991 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20260929 10:23:43.934020 28298 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677423, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.189@KRBTEST.COM for oryx/127.21.33.189@KRBTEST.COM
I20260929 10:23:43.942368 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.934491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.0.0.1:48800)
0929 10:23:43.934895 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.934906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.934972 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.935168 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.935170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.935296 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.935386 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.935390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.936413 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.936416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.936902 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.936909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.937385 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.937387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.937457 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.937866 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.937877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.939035 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.940909 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.940911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.940913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.941098 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.941446 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.941448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.941449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.941525 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.941928 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.941930 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.942005 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.942083 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.942181 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347}
I20260929 10:23:43.942368 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.934537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.21.33.189:45771)
0929 10:23:43.934623 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.934628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.934964 (+ 336us) server_negotiation.cc:407] Connection header received
0929 10:23:43.935045 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.935047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.935081 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.935144 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.935518 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.936321 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.937013 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.937281 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.937371 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.939170 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.939173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.939183 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.939203 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.940795 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.941255 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.941263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.941266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.941358 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.941649 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.941652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.941653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.941841 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.941929 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.942133 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.942249 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":24}
I20260929 10:23:43.945432 28298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Bootstrap starting.
I20260929 10:23:43.947667 28298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:43.948356 28298 log.cc:824] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:43.950352 28298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: No bootstrap required, opened a new log
I20260929 10:23:43.953296 28298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } }
I20260929 10:23:43.953504 28298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:43.953537 28298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b76f60281ac4622af2631809c3d238c, State: Initialized, Role: FOLLOWER
I20260929 10:23:43.953974 28298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } }
I20260929 10:23:43.954437 28304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } } }
I20260929 10:23:43.954617 28304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:23:43.955466 28298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 10:23:43.960351 28315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:23:43.960441 28315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:23:43.962365 28230 master_runner.cc:428] Detected that this master 0b76f60281ac4622af2631809c3d238c is joining an existing cluster
I20260929 10:23:43.962428 28230 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.33.189:45771
I20260929 10:23:43.972675 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.964462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.21.33.190:35965)
0929 10:23:43.964604 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.964609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.964883 (+ 274us) server_negotiation.cc:407] Connection header received
0929 10:23:43.965083 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.965087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.965139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.965253 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.965991 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.966518 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.967379 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.967607 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.967716 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.969005 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.969009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.969011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.969037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.970752 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.971336 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.971340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.971342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.971429 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.971718 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.971721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.971723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.971955 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.972027 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.972387 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.972512 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:23:43.972675 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.964772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.21.33.189:45771)
0929 10:23:43.964890 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.964895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.965007 (+ 112us) server_negotiation.cc:407] Connection header received
0929 10:23:43.965090 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.965093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.965138 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.965238 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.965797 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.966571 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.967742 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.968052 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.968121 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.969408 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.969412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.969416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.969443 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.971106 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.971495 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.971501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.971503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.971580 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.971842 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.971845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.971847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.972012 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.972115 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.972357 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.972514 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":50}
I20260929 10:23:43.972690 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.964523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.0.0.1:48804)
0929 10:23:43.964925 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.964936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.965018 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.965276 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.965279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.965542 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.965669 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.965677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.966712 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.966715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.967608 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.967618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.968253 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.968256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.968362 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.968799 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.968814 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.969287 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.971215 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.971217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.971219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.971404 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.971667 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.971671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.971673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.971754 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.972107 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.972110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.972211 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.972316 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.972465 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":100,"threads_started":1}
I20260929 10:23:43.973523 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.964410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55100)
0929 10:23:43.964791 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.964807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.964960 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.965408 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.965413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.965767 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.965881 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.965886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.966643 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.966646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.967251 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.967258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.967721 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.967724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.967812 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.968370 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.968390 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.968868 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.970903 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.970907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.970910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.971219 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.971536 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.971539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.971541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.971621 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.972097 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.972100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.972198 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.973171 (+ 973us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.973345 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":118,"threads_started":1}
Sep 29 10:23:43 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677422, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.190@KRBTEST.COM for oryx/127.21.33.189@KRBTEST.COM
I20260929 10:23:43.992106 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.978484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.21.33.189:45771)
0929 10:23:43.978592 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:43.978596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:43.981643 (+ 3047us) server_negotiation.cc:407] Connection header received
0929 10:23:43.981929 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:43.981934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:43.981998 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:43.982087 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:43.982885 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.983675 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.984912 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.985502 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.985599 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.988466 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:43.988470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:43.988473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:43.988500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:43.990125 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.990654 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.990657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.990659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.990757 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:43.991064 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:43.991067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:43.991069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:43.991285 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:43.991382 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:43.991822 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 10:23:43.991944 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:23:43.992257 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:43.978430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.0.0.1:48814)
0929 10:23:43.981506 (+ 3076us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:43.981565 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:43.981742 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:43.982133 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:43.982140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:43.982635 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:43.982768 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.982775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.983788 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.983791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.984789 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:43.984798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:43.985725 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:43.985740 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:43.985863 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:43.986446 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:43.986461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:43.988303 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:43.990240 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.990248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.990254 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.990550 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.990858 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:43.990861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:43.990862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:43.990963 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:43.991404 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:43.991414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:43.991588 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 10:23:43.991748 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 10:23:43.991976 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2891,"thread_start_us":104,"threads_started":1}
I20260929 10:23:43.994005 27766 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.33.189:45771: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b76f60281ac4622af2631809c3d238c"
member_type: NON_VOTER
last_known_addr {
host: "127.21.33.189"
port: 45771
}
attrs {
promote: true
}
}
dest_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c"
cas_config_opid_index: -1
I20260929 10:23:43.994715 27766 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.21.33.188@KRBTEST.COM; defaulting to no policy
I20260929 10:23:43.996733 28326 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0b76f60281ac4622af2631809c3d238c (127.21.33.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } }
I20260929 10:23:43.997298 28272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:43.999576 28326 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.33.189:45771
I20260929 10:23:43.999616 28272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Refusing update from remote peer d47f8e27fada4f2d8fb7199955d5eb1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.188@KRBTEST.COM" created.
I20260929 10:23:44.000289 28230 master.cc:537] Master@127.21.33.189:45771 shutting down...
I20260929 10:23:44.001426 28326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:44.000013 28325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } }
I20260929 10:23:44.002270 28325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
W20260929 10:23:44.004160 27752 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 10:23:44.005155 27752 consensus_peers.cc:487] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c -> Peer 0b76f60281ac4622af2631809c3d238c (127.21.33.189:45771): Couldn't send request to peer 0b76f60281ac4622af2631809c3d238c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 10:23:44.005785 28230 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:44.006068 28230 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:44.006170 28230 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: stopping tablet replica
I20260929 10:23:44.011528 28230 master.cc:552] Master@127.21.33.189:45771 shutdown complete.
I20260929 10:23:44.011583 28230 master_runner.cc:305] Clearing existing system tablet
I20260929 10:23:44.011835 28230 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.011890 28230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.013160 28230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:44.014551 28331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.014792 28230 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.014951 28230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:44.014998 28230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
uuid: "0b76f60281ac4622af2631809c3d238c"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "86a48b810704bb11979849f2a71e04dd"
server_key_iv: "12179135079cd415fad44c09922f06ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.017892 28230 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.188.keytab.
Entry for principal oryx/127.21.33.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.188.keytab.
I20260929 10:23:44.020700 28230 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 10:23:44.020810 28230 log.cc:1196] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 10:23:44.021070 28230 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Deleting consensus metadata
I20260929 10:23:44.021122 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:34991
--webserver_interface=127.21.33.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.188.keytab
--principal=oryx/127.21.33.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:23:44.021311 28230 master_runner.cc:315] Copying system tablet from 127.21.33.190:35965
I20260929 10:23:44.022502 28230 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.33.190:35965
I20260929 10:23:44.032461 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.023159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.21.33.190:35965)
0929 10:23:44.023309 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.023314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.023435 (+ 121us) server_negotiation.cc:407] Connection header received
0929 10:23:44.023602 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.023605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.023659 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.023753 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.024326 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.025150 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.026575 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.026930 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.027035 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.028612 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.028617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.028619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.028647 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.030466 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.030999 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.031003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.031005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.031094 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.031493 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.031497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.031499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.031719 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.031826 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.032162 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.032285 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:23:44.032585 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.023060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55116)
0929 10:23:44.023340 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.023358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.023493 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.023781 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.023785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.024024 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.024189 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.024198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.025328 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.025335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.026420 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.026435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.027062 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.027068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.027207 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.027839 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.027859 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.028409 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.030588 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.030591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.030594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.030887 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.031219 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.031225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.031230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.031340 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.031836 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.031841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.031972 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.032102 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.032294 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260929 10:23:44.033579 27786 tablet_copy_service.cc:140] P d47f8e27fada4f2d8fb7199955d5eb1c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b76f60281ac4622af2631809c3d238c ({username='oryx', principal='oryx/127.21.33.189@KRBTEST.COM'} at 127.0.0.1:55116)
I20260929 10:23:44.033778 27786 tablet_copy_service.cc:161] P d47f8e27fada4f2d8fb7199955d5eb1c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b76f60281ac4622af2631809c3d238c at {username='oryx', principal='oryx/127.21.33.189@KRBTEST.COM'} at 127.0.0.1:55116: session id = 0b76f60281ac4622af2631809c3d238c-00000000000000000000000000000000
I20260929 10:23:44.035576 27786 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 10:23:44.037328 28230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 10:23:44.039778 28230 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: tablet copy: Starting download of 0 data blocks...
I20260929 10:23:44.039971 28230 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: tablet copy: Starting download of 1 WAL segments...
I20260929 10:23:44.041859 28230 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 10:23:44.043599 27786 tablet_copy_service.cc:342] P d47f8e27fada4f2d8fb7199955d5eb1c: Request end of tablet copy session 0b76f60281ac4622af2631809c3d238c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.33.189@KRBTEST.COM'} at 127.0.0.1:55116
I20260929 10:23:44.043723 27786 tablet_copy_service.cc:433] P d47f8e27fada4f2d8fb7199955d5eb1c: ending tablet copy session 0b76f60281ac4622af2631809c3d238c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b76f60281ac4622af2631809c3d238c
I20260929 10:23:44.050442 28230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:44.052145 28339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.052145 28342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.052399 28230 server_base.cc:1034] running on GCE node
W20260929 10:23:44.052489 28340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.052673 28230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:44.052950 28230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:44.054090 28230 hybrid_clock.cc:648] HybridClock initialized: now 1790677424054076 us; error 29 us; skew 500 ppm
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:44.055752 28230 init.cc:377] Logged in from keytab as oryx/127.21.33.189@KRBTEST.COM (short username oryx)
I20260929 10:23:44.056214 28230 webserver.cc:460] Webserver started at http://127.21.33.189:42731/ using document root <none> and password file <none>
I20260929 10:23:44.056393 28230 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.056434 28230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.058125 28230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:44.059275 28349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.059517 28230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:44.059623 28230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
uuid: "0b76f60281ac4622af2631809c3d238c"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "86a48b810704bb11979849f2a71e04dd"
server_key_iv: "12179135079cd415fad44c09922f06ec"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.059741 28230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.067610 28230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:44.069823 28230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:44.074695 28230 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.189:45771
I20260929 10:23:44.075179 28401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.189:45771 every 8 connection(s)
I20260929 10:23:44.075380 28230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
I20260929 10:23:44.077613 28402 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:23:44.078647 28402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Bootstrap starting.
I20260929 10:23:44.089694 28402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:23:44.090049 28402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: Bootstrap complete.
I20260929 10:23:44.090634 28402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } }
I20260929 10:23:44.090772 28402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b76f60281ac4622af2631809c3d238c, State: Initialized, Role: LEARNER
I20260929 10:23:44.090873 28402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } }
I20260929 10:23:44.091240 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } }
I20260929 10:23:44.091331 28402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:44.091427 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: LEARNER
I20260929 10:23:44.094309 28416 catalog_manager.cc:1259] Loaded cluster ID: 3d8c5231345e459d85e898a741d21cdf
I20260929 10:23:44.094399 28416 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: loading cluster ID for follower catalog manager: success
I20260929 10:23:44.096432 28416 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: acquiring CA information for follower catalog manager: success
I20260929 10:23:44.097888 28416 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 10:23:44.152767 28337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:44.153069 28337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:44.153167 28337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:44.157486 28337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:44.157560 28337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:44.157594 28337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:44.157655 28337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:44.157707 28337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:44.163095 28337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.188.keytab
--principal=oryx/127.21.33.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.33.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:44.164139 28337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:44.165261 28337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:44.173404 28421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.173435 28422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.173435 28424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.173470 28337 server_base.cc:1034] running on GCE node
I20260929 10:23:44.174163 28337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:44.175275 28337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:44.176481 28337 hybrid_clock.cc:648] HybridClock initialized: now 1790677424176446 us; error 51 us; skew 500 ppm
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:44.179293 28337 init.cc:377] Logged in from keytab as oryx/127.21.33.188@KRBTEST.COM (short username oryx)
I20260929 10:23:44.180173 28337 webserver.cc:460] Webserver started at http://127.21.33.188:42009/ using document root <none> and password file <none>
I20260929 10:23:44.180734 28337 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.180830 28337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.181077 28337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:44.183180 28337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/instance:
uuid: "91a4661763b44194adbc81a50a6aea93"
format_stamp: "Formatted at 2026-09-29 10:23:44 on dist-test-slave-2rbl"
server_key: "ecd9773ec6d4aee24721e6a13ce054fb"
server_key_iv: "6ce9953d5093abe129860deec83a160b"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.183777 28337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal/instance:
uuid: "91a4661763b44194adbc81a50a6aea93"
format_stamp: "Formatted at 2026-09-29 10:23:44 on dist-test-slave-2rbl"
server_key: "ecd9773ec6d4aee24721e6a13ce054fb"
server_key_iv: "6ce9953d5093abe129860deec83a160b"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.187593 28337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 10:23:44.189965 28431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.191111 28337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:44.191258 28337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
uuid: "91a4661763b44194adbc81a50a6aea93"
format_stamp: "Formatted at 2026-09-29 10:23:44 on dist-test-slave-2rbl"
server_key: "ecd9773ec6d4aee24721e6a13ce054fb"
server_key_iv: "6ce9953d5093abe129860deec83a160b"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.191381 28337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.207254 28337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:44.210204 28337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:44.210376 28337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:44.218171 28337 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.188:34991
I20260929 10:23:44.218483 28483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.188:34991 every 8 connection(s)
I20260929 10:23:44.219336 28337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
I20260929 10:23:44.222143 28484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:44.225867 28484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:23:44.229034 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 28337
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.188@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.33.188@KRBTEST.COM
I20260929 10:23:44.240803 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.227779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.21.33.190:35965)
0929 10:23:44.227948 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.227953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.228720 (+ 767us) server_negotiation.cc:407] Connection header received
0929 10:23:44.230049 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.230052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.230113 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.230199 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.231726 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.232262 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.233524 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.233781 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.233891 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.236731 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.236735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.236738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.236764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.238540 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.239118 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.239124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.239129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.239219 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.239591 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.239594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.239596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.239834 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.239907 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.240478 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.240622 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:23:44.241693 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.228069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55126)
0929 10:23:44.228568 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.228620 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.229792 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.230538 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.230546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.231010 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.231555 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.231576 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.232403 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.232406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.233410 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.233420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.233925 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.233928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.234135 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.234789 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.234810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.236559 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.238645 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.238653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.238661 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.239018 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.239323 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.239326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.239328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.239499 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.239937 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.239948 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.240209 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.240400 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.240763 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":44,"thread_start_us":109,"threads_started":1}
I20260929 10:23:44.241694 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.229944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.21.33.188:34991)
0929 10:23:44.230317 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.230328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.230366 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:23:44.230511 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.230516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.230717 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.230899 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.231628 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.232722 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.233792 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.234271 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.234352 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.237181 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.237188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.237242 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.237271 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.239101 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.239596 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.239603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.239604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.239679 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.239989 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.239991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.239992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.240224 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.240402 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.240647 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.240824 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":2816,"thread_start_us":162,"threads_started":1}
I20260929 10:23:44.243840 28484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.188@KRBTEST.COM for oryx/127.21.33.189@KRBTEST.COM
I20260929 10:23:44.252449 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.244341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.0.0.1:48816)
0929 10:23:44.244458 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.244478 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.244537 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.245056 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.245059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.245283 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.245376 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.245381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.246435 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.246442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.247060 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.247068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.247591 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.247595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.247702 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.248252 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.248274 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.249654 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.251219 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.251222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.251223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.251407 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.251674 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.251676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.251678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.251755 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.252078 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.252080 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.252155 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.252232 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.252332 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 10:23:44.252604 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.244474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.21.33.189:45771)
0929 10:23:44.244763 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.244766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.244778 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:44.244810 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.244813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.244849 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.245047 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.245495 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.246178 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.247210 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.247469 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.247630 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.249779 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.249782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.249784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.249807 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.251124 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.251495 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.251498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.251499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.251572 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.251830 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.251833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.251834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.252003 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.252077 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.252334 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.252441 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
I20260929 10:23:44.253310 28484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.188@KRBTEST.COM for oryx/127.21.33.188@KRBTEST.COM
I20260929 10:23:44.260968 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.253778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.0.0.1:55682)
0929 10:23:44.253884 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.253899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.253972 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.254159 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.254161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.254289 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.254354 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.254359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.255097 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.255099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.255784 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.255793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.256257 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.256260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.256353 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.256879 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.256892 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.258125 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.259571 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.259574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.259575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.259755 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.260013 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.260018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.260020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.260100 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.260459 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.260462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.260547 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.260653 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.260791 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 10:23:44.260968 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.253821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.21.33.188:34991)
0929 10:23:44.253934 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.253937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.253949 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:44.254040 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.254042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.254074 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.254135 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.254445 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.254976 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.255876 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.256147 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.256226 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.258248 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.258251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.258253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.258272 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.259483 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.259836 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.259839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.259840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.259922 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.260194 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.260197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.260199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.260369 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.260470 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.260694 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.260847 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":45}
I20260929 10:23:44.264147 28484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Bootstrap starting.
I20260929 10:23:44.266446 28484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:44.267093 28484 log.cc:824] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:44.268033 27957 heartbeater.cc:503] Master 127.21.33.190:35965 was elected leader, sending a full tablet report...
I20260929 10:23:44.269049 28484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: No bootstrap required, opened a new log
I20260929 10:23:44.271968 28484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } }
I20260929 10:23:44.272159 28484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:44.272197 28484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a4661763b44194adbc81a50a6aea93, State: Initialized, Role: FOLLOWER
I20260929 10:23:44.272563 28484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } }
I20260929 10:23:44.272955 28490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } } }
I20260929 10:23:44.273068 28490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:23:44.273722 28484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 10:23:44.278110 28501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:23:44.278184 28501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:23:44.281282 28337 master_runner.cc:428] Detected that this master 91a4661763b44194adbc81a50a6aea93 is joining an existing cluster
I20260929 10:23:44.281339 28337 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.33.188:34991
I20260929 10:23:44.292228 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.283334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.0.0.1:48826)
0929 10:23:44.283773 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.283787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.283917 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.284278 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.284283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.284548 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.284673 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.284681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.285776 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.285779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.286667 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.286675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.287345 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.287350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.287473 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.288038 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.288052 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.288422 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.290638 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.290641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.290644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.290902 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.291206 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.291209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.291211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.291304 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.291675 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.291678 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.291785 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.291915 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.292044 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":24,"thread_start_us":82,"threads_started":1}
I20260929 10:23:44.292217 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.283409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.21.33.189:45771)
0929 10:23:44.283544 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.283548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.283855 (+ 307us) server_negotiation.cc:407] Connection header received
0929 10:23:44.284025 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.284029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.284083 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.284159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.284814 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.285643 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.286796 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.287126 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.287189 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.288774 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.288777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.288779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.288798 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.290526 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.291010 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.291015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.291020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.291107 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.291396 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.291399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.291400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.291592 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.291714 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.291974 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.292087 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:23:44.293644 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.284135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.21.33.190:35965)
0929 10:23:44.284375 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.284380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.284393 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:44.284432 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.284437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.284483 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.284570 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.285150 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.286002 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.287693 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.288039 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.288131 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.289405 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.289410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.289415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.289444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.291117 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.291832 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.291836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.291838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.291917 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.292193 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.292196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.292198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.292388 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.292455 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.293380 (+ 925us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.293496 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260929 10:23:44.294499 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.283297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55142)
0929 10:23:44.283711 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.283725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.283820 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.284609 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.284612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.284824 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.285010 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.285018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.286141 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.286144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.287554 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.287564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.288282 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.288287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.288382 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.288775 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.288790 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.289187 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.291412 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.291414 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.291417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.291712 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.292021 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.292024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.292026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.292102 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.293051 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.293054 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.293153 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.294153 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.294326 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":51,"thread_start_us":139,"threads_started":1}
I20260929 10:23:44.295109 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27828
I20260929 10:23:44.302556 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:35209
--local_ip_for_outbound_sockets=127.21.33.129
--tserver_master_addrs=127.21.33.190:35965
--webserver_port=34671
--webserver_interface=127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab
--principal=oryx/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677422, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.190@KRBTEST.COM for oryx/127.21.33.188@KRBTEST.COM
I20260929 10:23:44.310022 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.284246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.21.33.188:34991)
0929 10:23:44.284703 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.284710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.284723 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:44.284754 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.284757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.284801 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.284864 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.285341 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.286022 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.287010 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.287307 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.287414 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.306365 (+ 18951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.306373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.306380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.306413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.308102 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.308569 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.308572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.308574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.308644 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.308946 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.308952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.308957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.309183 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.309308 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.309670 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.309830 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":396}
I20260929 10:23:44.310530 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.283761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.0.0.1:55688)
0929 10:23:44.284146 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.284159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.284231 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.284911 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.284913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.285103 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.285226 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.285242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.286122 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.286125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.286919 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.286926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.287515 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.287519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.287616 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.305739 (+ 18123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.305755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.306214 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.308224 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.308226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.308242 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.308469 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.308752 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.308756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.308759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.308849 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.309378 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.309381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.309487 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.309775 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.310340 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260929 10:23:44.311944 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.299726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.0.0.1:55696)
0929 10:23:44.299853 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.299869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.299947 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.302589 (+ 2642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.302592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.302823 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.302963 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.302973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.304062 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.304067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.304983 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.304996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.305975 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.305981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.306077 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.306555 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.306573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.308205 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.310409 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.310412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.310415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.310661 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.310974 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.310977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.310978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.311065 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.311473 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.311476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.311570 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.311659 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.311836 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 10:23:44.312486 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.299769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.21.33.188:34991)
0929 10:23:44.302307 (+ 2538us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.302316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.302329 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:44.302397 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.302400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.302451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.302540 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.303104 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.303942 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.305097 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.305830 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.308328 (+ 2498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.308455 (+ 127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.308458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.308461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.308478 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.310288 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.310788 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.310791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.310792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.310867 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.311170 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.311172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.311174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.311362 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.312136 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.312216 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.312326 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2452,"thread_start_us":124,"threads_started":1}
I20260929 10:23:44.313299 27766 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.33.188:34991: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91a4661763b44194adbc81a50a6aea93"
member_type: NON_VOTER
last_known_addr {
host: "127.21.33.188"
port: 34991
}
attrs {
promote: true
}
}
dest_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c"
cas_config_opid_index: 5
I20260929 10:23:44.313885 27766 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } }
I20260929 10:23:44.314939 28325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91a4661763b44194adbc81a50a6aea93 (127.21.33.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.315454 28458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:44.316285 28325 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.33.188:34991
I20260929 10:23:44.316455 28325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.316529 28325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:44.317350 28458 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Refusing update from remote peer d47f8e27fada4f2d8fb7199955d5eb1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 10:23:44.318012 28326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:44.318126 28337 master.cc:537] Master@127.21.33.188:34991 shutting down...
W20260929 10:23:44.321115 27751 consensus_peers.cc:487] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c -> Peer 91a4661763b44194adbc81a50a6aea93 (127.21.33.188:34991): Couldn't send request to peer 91a4661763b44194adbc81a50a6aea93. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 10:23:44.321442 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.315028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.0.0.1:48832)
0929 10:23:44.315392 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.315406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.315478 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.317045 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.317048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.317065 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:44.317144 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.317149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.318147 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.318151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.319179 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.319188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.320216 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.320220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.320306 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.320359 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.320436 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.321354 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260929 10:23:44.321918 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.315262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.21.33.189:45771)
0929 10:23:44.316761 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.316767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.316783 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:44.316835 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.316838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.316911 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.317012 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:44.317315 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.318008 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.319309 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.319892 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.319992 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.320151 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.321715 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.321800 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406}
I20260929 10:23:44.322381 28368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 LEARNER]: Refusing update from remote peer d47f8e27fada4f2d8fb7199955d5eb1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 10:23:44.322876 28326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 10:23:44.323603 28337 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:44.323871 28337 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:44.323971 28337 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: stopping tablet replica
I20260929 10:23:44.324753 28368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91a4661763b44194adbc81a50a6aea93 (127.21.33.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.327163 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47f8e27fada4f2d8fb7199955d5eb1c. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.327257 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: LEARNER
I20260929 10:23:44.327325 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: NON_VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: true } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.327378 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: LEARNER
I20260929 10:23:44.327710 28325 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: attempting to promote NON_VOTER 0b76f60281ac4622af2631809c3d238c to VOTER
I20260929 10:23:44.328209 28325 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } }
I20260929 10:23:44.329725 28368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 LEARNER]: Refusing update from remote peer d47f8e27fada4f2d8fb7199955d5eb1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 10:23:44.330068 28326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 10:23:44.330240 27751 consensus_peers.cc:487] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c -> Peer 91a4661763b44194adbc81a50a6aea93 (127.21.33.188:34991): Couldn't send request to peer 91a4661763b44194adbc81a50a6aea93. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 10:23:44.332082 28326 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0b76f60281ac4622af2631809c3d238c (127.21.33.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.333323 28326 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.333436 28326 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:44.333900 28368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0b76f60281ac4622af2631809c3d238c (127.21.33.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.335155 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.335269 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:23:44.333539 28514 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.335515 28514 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:44.343575 28337 master.cc:552] Master@127.21.33.188:34991 shutdown complete.
I20260929 10:23:44.343648 28337 master_runner.cc:305] Clearing existing system tablet
I20260929 10:23:44.343895 28337 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.343967 28337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.345046 28337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 10:23:44.345912 28516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.346151 28337 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.346277 28337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:44.346320 28337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
uuid: "91a4661763b44194adbc81a50a6aea93"
format_stamp: "Formatted at 2026-09-29 10:23:44 on dist-test-slave-2rbl"
server_key: "ecd9773ec6d4aee24721e6a13ce054fb"
server_key_iv: "6ce9953d5093abe129860deec83a160b"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.348007 28337 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:44.349915 28337 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 10:23:44.350018 28337 log.cc:1196] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 10:23:44.350260 28337 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Deleting consensus metadata
I20260929 10:23:44.350410 28337 master_runner.cc:315] Copying system tablet from 127.21.33.190:35965
I20260929 10:23:44.351536 28337 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: tablet copy: Beginning tablet copy session from remote peer at address 127.21.33.190:35965
I20260929 10:23:44.359967 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.352096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55146)
0929 10:23:44.352412 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.352425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.352557 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.352824 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.352827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.353035 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.353145 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.353151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.353970 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.353975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.354634 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.354644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.355167 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.355171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.355334 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.355933 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.355947 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.356294 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.358269 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.358275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.358277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.358520 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.358813 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.358817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.358819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.358906 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.359386 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.359389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.359489 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.359619 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.359762 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
I20260929 10:23:44.359992 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.352164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.21.33.190:35965)
0929 10:23:44.352309 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.352315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.352510 (+ 195us) server_negotiation.cc:407] Connection header received
0929 10:23:44.352657 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.352660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.352720 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.352807 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.353320 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.353857 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.354816 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.355049 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.355177 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.356497 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.356501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.356504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.356533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.358168 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.358624 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.358628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.358630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.358718 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.359004 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.359007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.359009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.359279 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.359398 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.359673 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.359798 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:23:44.360816 27786 tablet_copy_service.cc:140] P d47f8e27fada4f2d8fb7199955d5eb1c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91a4661763b44194adbc81a50a6aea93 ({username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:55146)
I20260929 10:23:44.360932 27786 tablet_copy_service.cc:161] P d47f8e27fada4f2d8fb7199955d5eb1c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91a4661763b44194adbc81a50a6aea93 at {username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:55146: session id = 91a4661763b44194adbc81a50a6aea93-00000000000000000000000000000000
I20260929 10:23:44.361840 27786 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 10:23:44.363018 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 10:23:44.364854 28337 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: tablet copy: Starting download of 0 data blocks...
I20260929 10:23:44.365072 28337 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: tablet copy: Starting download of 1 WAL segments...
I20260929 10:23:44.366461 28337 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 10:23:44.368036 27786 tablet_copy_service.cc:342] P d47f8e27fada4f2d8fb7199955d5eb1c: Request end of tablet copy session 91a4661763b44194adbc81a50a6aea93-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:55146
I20260929 10:23:44.368140 27786 tablet_copy_service.cc:433] P d47f8e27fada4f2d8fb7199955d5eb1c: ending tablet copy session 91a4661763b44194adbc81a50a6aea93-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91a4661763b44194adbc81a50a6aea93
I20260929 10:23:44.375371 28337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:44.377102 28523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.377182 28526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.378033 28337 server_base.cc:1034] running on GCE node
W20260929 10:23:44.381310 28524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.381547 28337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:44.381886 28337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:44.383020 28337 hybrid_clock.cc:648] HybridClock initialized: now 1790677424383035 us; error 48 us; skew 500 ppm
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:44.384527 28337 init.cc:377] Logged in from keytab as oryx/127.21.33.188@KRBTEST.COM (short username oryx)
I20260929 10:23:44.385020 28337 webserver.cc:460] Webserver started at http://127.21.33.188:35059/ using document root <none> and password file <none>
I20260929 10:23:44.385274 28337 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.385324 28337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.386442 28337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:44.387399 28533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.387625 28337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:44.387698 28337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
uuid: "91a4661763b44194adbc81a50a6aea93"
format_stamp: "Formatted at 2026-09-29 10:23:44 on dist-test-slave-2rbl"
server_key: "ecd9773ec6d4aee24721e6a13ce054fb"
server_key_iv: "6ce9953d5093abe129860deec83a160b"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.387799 28337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.403968 28337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:44.406112 28337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:44.411109 28337 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.188:34991
I20260929 10:23:44.411767 28337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
I20260929 10:23:44.412156 28585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.188:34991 every 8 connection(s)
I20260929 10:23:44.413411 28586 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:23:44.414299 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Bootstrap starting.
I20260929 10:23:44.422943 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:23:44.423269 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Bootstrap complete.
I20260929 10:23:44.423789 28586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } }
I20260929 10:23:44.423929 28586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91a4661763b44194adbc81a50a6aea93, State: Initialized, Role: LEARNER
I20260929 10:23:44.424047 28586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } }
I20260929 10:23:44.424386 28589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.424558 28589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: This master's current role is: LEARNER
I20260929 10:23:44.424566 28586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:44.428526 28600 catalog_manager.cc:1259] Loaded cluster ID: 3d8c5231345e459d85e898a741d21cdf
I20260929 10:23:44.428578 28600 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: loading cluster ID for follower catalog manager: success
I20260929 10:23:44.430451 28600 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: acquiring CA information for follower catalog manager: success
I20260929 10:23:44.431552 28600 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 10:23:44.467746 28511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:44.468071 28511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:44.468137 28511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:44.474339 28511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:44.474414 28511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:44.474502 28511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:44.482590 28511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab
--principal=oryx/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:35209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=34671
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:44.483788 28511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:44.484989 28511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:23:44.485819 28091 heartbeater.cc:503] Master 127.21.33.190:35965 was elected leader, sending a full tablet report...
W20260929 10:23:44.496218 28605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.496284 28608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.496260 28511 server_base.cc:1034] running on GCE node
W20260929 10:23:44.496224 28606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.496830 28511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:44.497550 28511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:44.498739 28511 hybrid_clock.cc:648] HybridClock initialized: now 1790677424498718 us; error 29 us; skew 500 ppm
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:44.501709 28511 init.cc:377] Logged in from keytab as oryx/127.21.33.129@KRBTEST.COM (short username oryx)
I20260929 10:23:44.502549 28511 webserver.cc:460] Webserver started at http://127.21.33.129:34671/ using document root <none> and password file <none>
I20260929 10:23:44.503103 28511 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.503221 28511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.507440 28511 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:44.509513 28615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.510694 28511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:23:44.510828 28511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "2cb6f3e06f364173bc2e0715ddec5754"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "9e377d8dfce20a15e1239f622dbdeb95"
server_key_iv: "90cb9c0049423afbec8059eff024389e"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.511425 28511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.529594 28511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:44.532836 28511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:44.533036 28511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:44.533638 28511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:44.534458 28511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:44.534528 28511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.534597 28511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:44.534644 28511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.543303 28511 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:35209
I20260929 10:23:44.543345 28728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:35209 every 8 connection(s)
I20260929 10:23:44.544309 28511 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:44.551751 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 28511
I20260929 10:23:44.551959 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27962
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.188@KRBTEST.COM
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.189@KRBTEST.COM
I20260929 10:23:44.560036 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:35193
--local_ip_for_outbound_sockets=127.21.33.130
--tserver_master_addrs=127.21.33.190:35965
--webserver_port=38753
--webserver_interface=127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab
--principal=oryx/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:23:44.561878 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.546121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:56397 (local address 127.21.33.189:45771)
0929 10:23:44.546344 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.546350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.547132 (+ 782us) server_negotiation.cc:407] Connection header received
0929 10:23:44.548311 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.548315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.548372 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.548476 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.550005 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.550531 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.552884 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.553586 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.553670 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.556928 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.556934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.556936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.556964 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.558962 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.559818 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.559822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.559824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.559911 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.560387 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.560395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.560397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.560637 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.560739 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.561577 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.561697 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 10:23:44.561957 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.546093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:33139 (local address 127.21.33.190:35965)
0929 10:23:44.546225 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.546230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.547134 (+ 904us) server_negotiation.cc:407] Connection header received
0929 10:23:44.548441 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.548444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.548489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.548582 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.550032 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.550834 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.552052 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.552487 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.552581 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.556682 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.556689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.556695 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.556731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.559515 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.560045 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.560052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.560057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.560149 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.560828 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.560835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.560839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.561085 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.561162 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.561602 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.561750 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:23:44.562942 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.547533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.21.33.129:34547)
0929 10:23:44.547837 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.547850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.548112 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.548771 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.548775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.549282 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.549870 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.549885 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.551184 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.551188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.551968 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.551978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.552745 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.552752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.552962 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.553607 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.553631 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.556606 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.559295 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.559298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.559300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.559679 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.560022 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.560025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.560027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.560261 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.560785 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.560795 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.561271 (+ 476us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.561513 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.561852 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":161,"spinlock_wait_cycles":17408,"thread_start_us":71,"threads_started":1}
I20260929 10:23:44.562952 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.546525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:33139)
0929 10:23:44.546977 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.547031 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.548110 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.548643 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.548653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.549282 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.549871 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.549885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.550970 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.550973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.551902 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.551914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.553443 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.553450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.553595 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.554088 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.554107 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.556494 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.559676 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.559678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.559680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.559923 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.560557 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.560563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.560568 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.560667 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.561269 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.561275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.561389 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.562514 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.562691 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":113,"spinlock_wait_cycles":30464,"thread_start_us":148,"threads_started":1}
I20260929 10:23:44.563220 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.546525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.21.33.129:56397)
0929 10:23:44.546977 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.547030 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.548111 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.548644 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.548652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.549281 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.549868 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.549882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.550671 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.550674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.551926 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.551936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.554123 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.554127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.554232 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.554701 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.554794 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.556783 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.559128 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.559137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.559158 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.559684 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.560022 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.560025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.560027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.560261 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.560838 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.560841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.561271 (+ 430us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.562132 (+ 861us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.562268 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":142,"spinlock_wait_cycles":44928,"thread_start_us":136,"threads_started":1}
I20260929 10:23:44.563621 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.547670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:34547 (local address 127.21.33.188:34991)
0929 10:23:44.547903 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.547906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.547919 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:44.548605 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.548609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.548662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.548751 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.550032 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.551069 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.552091 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.552465 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.552571 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.556759 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.556762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.556765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.556792 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.559165 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.559818 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.559822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.559824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.559911 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.560387 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.560395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.560396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.560648 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.560729 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.563289 (+ 2560us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.563430 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260929 10:23:44.564409 28731 heartbeater.cc:347] Connected to a master server at 127.21.33.189:45771
I20260929 10:23:44.564463 28729 heartbeater.cc:347] Connected to a master server at 127.21.33.188:34991
I20260929 10:23:44.564503 28730 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35965
I20260929 10:23:44.564646 28729 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.564646 28731 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.564646 28730 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.565266 28730 heartbeater.cc:511] Master 127.21.33.190:35965 requested a full tablet report, sending...
I20260929 10:23:44.565263 28731 heartbeater.cc:511] Master 127.21.33.189:45771 requested a full tablet report, sending...
I20260929 10:23:44.565263 28729 heartbeater.cc:511] Master 127.21.33.188:34991 requested a full tablet report, sending...
I20260929 10:23:44.566695 27766 ts_manager.cc:194] Re-registered known tserver with Master: 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:44.566839 28550 ts_manager.cc:194] Registered new tserver with Master: 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:44.566849 28358 ts_manager.cc:194] Registered new tserver with Master: 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:44.567691 27766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:33139
W20260929 10:23:44.689379 28737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:44.689817 28737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:44.689904 28737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:44.694648 28737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:44.694734 28737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:44.694862 28737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:44.697031 28225 heartbeater.cc:503] Master 127.21.33.190:35965 was elected leader, sending a full tablet report...
I20260929 10:23:44.702847 28737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab
--principal=oryx/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:35193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=38753
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:44.704241 28737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:44.705463 28737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:44.715327 28742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.715385 28745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.715406 28743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.715732 28737 server_base.cc:1034] running on GCE node
I20260929 10:23:44.716188 28737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:44.716814 28737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:44.717998 28737 hybrid_clock.cc:648] HybridClock initialized: now 1790677424717979 us; error 32 us; skew 500 ppm
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:44.720705 28737 init.cc:377] Logged in from keytab as oryx/127.21.33.130@KRBTEST.COM (short username oryx)
I20260929 10:23:44.721664 28737 webserver.cc:460] Webserver started at http://127.21.33.130:38753/ using document root <none> and password file <none>
I20260929 10:23:44.722227 28737 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.722334 28737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.726209 28737 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:44.728174 28752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.729487 28737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:44.729612 28737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "0f177df46dbe4c1fbef83392d6da2f2d"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "27dc5b52fe5d46afbe9a882dc34da2de"
server_key_iv: "22de473718ea999cc21fb308514c9ac1"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.730146 28737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.751610 28737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:44.754628 28737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:44.754984 28737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:44.755865 28737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:44.756767 28737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:44.756848 28737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.756918 28737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:44.756948 28737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.766611 28737 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:35193
I20260929 10:23:44.766633 28865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:35193 every 8 connection(s)
I20260929 10:23:44.767598 28737 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:44.769059 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 28737
I20260929 10:23:44.769276 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 28096
I20260929 10:23:44.776484 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:46599
--local_ip_for_outbound_sockets=127.21.33.131
--tserver_master_addrs=127.21.33.190:35965
--webserver_port=40823
--webserver_interface=127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab
--principal=oryx/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.189@KRBTEST.COM
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.188@KRBTEST.COM
I20260929 10:23:44.786021 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.771503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:37789 (local address 127.21.33.190:35965)
0929 10:23:44.771640 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.771646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.773186 (+ 1540us) server_negotiation.cc:407] Connection header received
0929 10:23:44.773410 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.773414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.773464 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.773516 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.774260 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.775071 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.777155 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.777471 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.777583 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.781478 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.781484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.781489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.781523 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.783331 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784077 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.784081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.784083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.784194 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784609 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.784612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.784614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.784846 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.784932 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.785728 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.785855 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:44.786021 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.769398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:57863 (local address 127.21.33.189:45771)
0929 10:23:44.769886 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.769893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.771912 (+ 2019us) server_negotiation.cc:407] Connection header received
0929 10:23:44.772113 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.772119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.772180 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.772250 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.774265 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.775145 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.776757 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.777154 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.777284 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.781022 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.781026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.781029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.781058 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.783327 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784077 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.784081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.784083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.784186 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784609 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.784612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.784614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.784855 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.784923 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.785732 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.785855 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380}
I20260929 10:23:44.786149 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.769895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:35519 (local address 127.21.33.188:34991)
0929 10:23:44.770013 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.770018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.770741 (+ 723us) server_negotiation.cc:407] Connection header received
0929 10:23:44.772150 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.772157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.772235 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.772294 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.774256 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.775071 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.778832 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.779231 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.779316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.781988 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.781994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.781998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.782032 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.783865 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784427 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.784434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.784439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.784532 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784897 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.784904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.784907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.785155 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.785306 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.785770 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.785926 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:23:44.788002 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.771150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:37789)
0929 10:23:44.773085 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.773102 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.773284 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.773651 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.773654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.773838 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.774083 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.774104 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.775257 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.775261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.777017 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.777031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.777587 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.777591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.777682 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.778352 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.778377 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.781318 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.783461 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.783467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.783483 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.783960 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.784301 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.784306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.784496 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.784995 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.785002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.785425 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.786285 (+ 860us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.786734 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1869,"mutex_wait_us":307,"spinlock_wait_cycles":27648,"thread_start_us":86,"threads_started":1}
I20260929 10:23:44.788002 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.769844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.21.33.130:35519)
0929 10:23:44.770576 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.770634 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.771902 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.772648 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.772654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.773144 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.774096 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.774104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.775257 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.775261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.778685 (+ 3424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.778697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.779481 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.779484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.779602 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.780158 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.780177 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.781803 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.784002 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.784009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.784299 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.784656 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.784664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.784779 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.785376 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.785382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.785542 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.786353 (+ 811us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.786875 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":2427,"spinlock_wait_cycles":42752,"thread_start_us":684,"threads_started":1}
I20260929 10:23:44.788830 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.770116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.21.33.130:57863)
0929 10:23:44.771821 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.771837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.771985 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.772650 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.772658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.773146 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.774094 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.774108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.775305 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.775311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.776620 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.776628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.777325 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.777331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.777553 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.778454 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.778469 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.780848 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.783461 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.783467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.783483 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.783960 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.784306 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.784309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.784311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.784496 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.784995 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.785002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.785425 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.786649 (+ 1224us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.786784 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1622,"mutex_wait_us":227,"spinlock_wait_cycles":30464,"thread_start_us":134,"threads_started":1}
I20260929 10:23:44.789636 28867 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35965
I20260929 10:23:44.789719 28866 heartbeater.cc:347] Connected to a master server at 127.21.33.188:34991
I20260929 10:23:44.789783 28868 heartbeater.cc:347] Connected to a master server at 127.21.33.189:45771
I20260929 10:23:44.789919 28867 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.789919 28868 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.789919 28866 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.790551 28867 heartbeater.cc:511] Master 127.21.33.190:35965 requested a full tablet report, sending...
I20260929 10:23:44.790552 28868 heartbeater.cc:511] Master 127.21.33.189:45771 requested a full tablet report, sending...
I20260929 10:23:44.791158 28866 heartbeater.cc:511] Master 127.21.33.188:34991 requested a full tablet report, sending...
I20260929 10:23:44.791672 28358 ts_manager.cc:194] Registered new tserver with Master: 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193)
I20260929 10:23:44.791903 28550 ts_manager.cc:194] Registered new tserver with Master: 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193)
I20260929 10:23:44.792099 27766 ts_manager.cc:194] Re-registered known tserver with Master: 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193)
I20260929 10:23:44.792861 27766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:37789
W20260929 10:23:44.892159 28873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:44.892501 28873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:44.892584 28873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:44.896621 28873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:44.896721 28873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:44.896817 28873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:44.902567 28873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab
--principal=oryx/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:46599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=40823
--tserver_master_addrs=127.21.33.190:35965,127.21.33.189:45771,127.21.33.188:34991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:44.903653 28873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:44.904717 28873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:44.914664 28878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:44.914673 28881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.914873 28873 server_base.cc:1034] running on GCE node
W20260929 10:23:44.914644 28879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:44.915537 28873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:44.916150 28873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:44.917281 28873 hybrid_clock.cc:648] HybridClock initialized: now 1790677424917256 us; error 31 us; skew 500 ppm
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:44.920042 28873 init.cc:377] Logged in from keytab as oryx/127.21.33.131@KRBTEST.COM (short username oryx)
I20260929 10:23:44.920881 28873 webserver.cc:460] Webserver started at http://127.21.33.131:40823/ using document root <none> and password file <none>
I20260929 10:23:44.921466 28873 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:44.921541 28873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:44.925689 28873 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:44.928103 28889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.929352 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.925371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.21.33.188:34991)
0929 10:23:44.925546 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.925551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.925620 (+ 69us) server_negotiation.cc:407] Connection header received
0929 10:23:44.925794 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.925797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.925861 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.925976 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:44.926267 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.926836 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.928196 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.928824 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.928909 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.929022 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.929146 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.929241 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 10:23:44.929380 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.925245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.0.0.1:55704)
0929 10:23:44.925528 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.925543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.925675 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.926002 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.926008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.926036 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:44.926162 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.926169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.926948 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.926952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.928068 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.928079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.928927 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.928931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.929021 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.929047 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.929127 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.929237 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260929 10:23:44.929705 28873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:23:44.929975 28560 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 10:23:44.929818 28873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "5ccdcd9e0d9a437e9678399ebd420b5e"
format_stamp: "Formatted at 2026-09-29 10:23:43 on dist-test-slave-2rbl"
server_key: "7849226ad648b2a5900f55904974dc86"
server_key_iv: "f3804d5678fc027d14b812801ecf0217"
server_key_version: "encryptionkey@0"
I20260929 10:23:44.930401 28890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47f8e27fada4f2d8fb7199955d5eb1c. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:44.930517 28873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:44.930624 28890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: This master's current role is: LEARNER
I20260929 10:23:44.956691 28873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:44.960280 28873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:44.960485 28873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:44.961156 28873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:44.962019 28873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:44.962064 28873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.962126 28873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:44.962157 28873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:44.971066 28873 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:46599
I20260929 10:23:44.971092 29003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:46599 every 8 connection(s)
I20260929 10:23:44.972106 28873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:44.973918 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 28873
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.189@KRBTEST.COM
I20260929 10:23:44.988415 28488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.974419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:33853 (local address 127.21.33.189:45771)
0929 10:23:44.974543 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.974548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.975879 (+ 1331us) server_negotiation.cc:407] Connection header received
0929 10:23:44.976245 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.976248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.976304 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.976384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.977793 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.978329 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.979495 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.979766 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.979860 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.983861 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.983865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.983868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.983894 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.986513 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.987029 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.987033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.987035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.987118 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.987411 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.987415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.987416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.987613 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.987678 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.988146 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.988259 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:23:44.988592 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.973941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:34577 (local address 127.21.33.190:35965)
0929 10:23:44.974100 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.974107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.975115 (+ 1008us) server_negotiation.cc:407] Connection header received
0929 10:23:44.976283 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.976287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.976339 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.976390 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.977799 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.978600 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.979767 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.980072 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.980172 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.983857 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.983861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.983864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.983894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.986033 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.986670 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.986673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.986677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.986758 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.987163 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.987166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.987168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.987415 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.987503 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.988261 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.988398 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:23:44.989490 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.974452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:34577)
0929 10:23:44.974961 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.975011 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.976053 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.976556 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.976563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.977134 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.977643 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.977656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.978736 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.978739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.979646 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.979655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.980296 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.980301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.980402 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.980976 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.981001 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.983660 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.986166 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.986172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.986188 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.986559 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.986869 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.986873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.986875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.987055 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.987553 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.987560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.987873 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.988083 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.988413 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"spinlock_wait_cycles":32384,"thread_start_us":163,"threads_started":1}
I20260929 10:23:44.989490 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.974451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.21.33.131:33853)
0929 10:23:44.975775 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.975787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.976054 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.976555 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.976562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.977051 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.977642 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.977655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.978475 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.978478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.979361 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.979370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.980013 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.980017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.980201 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.981011 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.981022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.983660 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.986634 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.986639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.986641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.986926 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.987230 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.987233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.987235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.987309 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.987758 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:44.987760 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:44.987874 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 10:23:44.988600 (+ 726us) client_negotiation.cc:240] Negotiation successful
0929 10:23:44.988719 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":28,"spinlock_wait_cycles":45440,"thread_start_us":148,"threads_started":1}
I20260929 10:23:44.990796 29006 heartbeater.cc:347] Connected to a master server at 127.21.33.189:45771
I20260929 10:23:44.991008 29005 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35965
I20260929 10:23:44.991055 29006 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.991076 29005 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:44.991627 29006 heartbeater.cc:511] Master 127.21.33.189:45771 requested a full tablet report, sending...
I20260929 10:23:44.991627 29005 heartbeater.cc:511] Master 127.21.33.190:35965 requested a full tablet report, sending...
I20260929 10:23:44.992774 27766 ts_manager.cc:194] Re-registered known tserver with Master: 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131:46599)
I20260929 10:23:44.992902 28358 ts_manager.cc:194] Registered new tserver with Master: 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131:46599)
I20260929 10:23:44.993508 27766 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:34577
Sep 29 10:23:44 dist-test-slave-2rbl krb5kdc[27721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677424, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.188@KRBTEST.COM
I20260929 10:23:44.998654 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.980116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.21.33.189:45771)
0929 10:23:44.985442 (+ 5326us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.985446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.985457 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:23:44.985490 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.985493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.985534 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.985613 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.987149 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.989583 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.991222 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.991560 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.991649 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.993884 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.993887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.993890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.993912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.996197 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.996849 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.996853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.996857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.996948 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.997312 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.997318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.997323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.997570 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:44.997698 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:44.998299 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 10:23:44.998452 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5258,"thread_start_us":58,"threads_started":1}
I20260929 10:23:45.001241 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.975527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.21.33.131:53845)
0929 10:23:44.980751 (+ 5224us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:44.980765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:44.980864 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:44.982209 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:44.982211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:44.982411 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:44.983139 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.983149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.984195 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.984198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.985057 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:44.985065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.989559 (+ 4494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.989563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:44.989647 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.990118 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:44.990133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:44.996424 (+ 6291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:44.999437 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.999440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.999442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:44.999701 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:44.999983 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:44.999987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:44.999990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:45.000137 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:45.000611 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:45.000614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:45.000712 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 10:23:45.000858 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 10:23:45.001025 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5164,"thread_start_us":75,"threads_started":1}
I20260929 10:23:45.001183 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.976821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:53845 (local address 127.21.33.188:34991)
0929 10:23:44.976937 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.976941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.980842 (+ 3901us) server_negotiation.cc:407] Connection header received
0929 10:23:44.981964 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.981968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.982026 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.982079 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.983282 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.984063 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.985173 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.985577 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.985684 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.996591 (+ 10907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.996596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.996599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.996627 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.998978 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:44.999786 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:44.999794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:44.999796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:44.999888 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.000234 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.000238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.000242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.000481 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:45.000566 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.000907 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.001029 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:23:45.001758 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.981769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.21.33.188:34991)
0929 10:23:44.982119 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.982124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.989103 (+ 6979us) server_negotiation.cc:407] Connection header received
0929 10:23:44.989142 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.989147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.989217 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.989300 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.992288 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.993095 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.994293 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.994636 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.994745 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.996803 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.996807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.996810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.996838 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.999637 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.000107 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.000110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.000112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.000191 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.000471 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.000475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.000476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.000663 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:45.000793 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.001259 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.001614 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":68,"threads_started":1}
I20260929 10:23:45.001850 29004 heartbeater.cc:347] Connected to a master server at 127.21.33.188:34991
I20260929 10:23:45.001947 29004 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:45.002005 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:44.980486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.21.33.190:35965)
0929 10:23:44.985305 (+ 4819us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:44.985309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:44.986886 (+ 1577us) server_negotiation.cc:407] Connection header received
0929 10:23:44.986928 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:44.986932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:44.986975 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:44.987046 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:44.989492 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.992477 (+ 2985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.995164 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:44.995493 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:44.995608 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:44.997329 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:44.997332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:44.997334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:44.997382 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:44.999696 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.000362 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.000365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.000366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.000469 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.000752 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.000755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.000757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.000958 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:45.001080 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.001675 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.001815 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4735,"thread_start_us":54,"threads_started":1}
I20260929 10:23:45.002166 29004 heartbeater.cc:511] Master 127.21.33.188:34991 requested a full tablet report, sending...
I20260929 10:23:45.002923 28549 ts_manager.cc:194] Registered new tserver with Master: 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131:46599)
I20260929 10:23:45.004736 27766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:45.007756 27766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:45.010398 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: NON_VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: true } } }
I20260929 10:23:45.010536 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:45.013831 29022 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: attempting to promote NON_VOTER 91a4661763b44194adbc81a50a6aea93 to VOTER
I20260929 10:23:45.014192 29022 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:45.015316 28368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Refusing update from remote peer d47f8e27fada4f2d8fb7199955d5eb1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 10:23:45.015834 28560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 LEARNER]: Refusing update from remote peer d47f8e27fada4f2d8fb7199955d5eb1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 10:23:45.015967 29023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 10:23:45.016218 28514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 10:23:45.017541 29024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 91a4661763b44194adbc81a50a6aea93 (127.21.33.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.017931 28368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 91a4661763b44194adbc81a50a6aea93 (127.21.33.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.018704 28560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 91a4661763b44194adbc81a50a6aea93 (127.21.33.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.019308 29022 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.019338 29024 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.019461 29024 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:45.019416 29022 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:45.019699 29032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.019832 29032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:23:45.020462 28890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.020599 28890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:23:45.021394 29023 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:45.021500 29023 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:45.029543 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.024367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:35193 (local address 127.0.0.1:51504)
0929 10:23:45.024464 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:45.024479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:45.024552 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:45.025495 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:45.025500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:45.025516 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:45.025619 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.025624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.027117 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.027121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.028070 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.028079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.029090 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.029093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.029227 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.029253 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:23:45.029319 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 10:23:45.029404 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":4}
I20260929 10:23:45.029587 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.024539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.21.33.130:35193)
0929 10:23:45.024898 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.024911 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.024936 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:23:45.025062 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.025068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.025334 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.025824 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:45.025959 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.027020 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.028201 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.028981 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.029084 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.029291 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.029354 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.029449 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":130,"threads_started":1}
I20260929 10:23:45.030552 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.024893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35209 (local address 127.0.0.1:58220)
0929 10:23:45.025362 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:45.025375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:45.025461 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:45.026202 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:45.026205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:45.026221 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:45.026324 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.026331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.028044 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.028048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.028835 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.028843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.030148 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.030152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.030248 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.030272 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:23:45.030356 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 10:23:45.030419 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":7,"thread_start_us":100,"threads_started":1}
I20260929 10:23:45.030915 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.025118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:46599 (local address 127.0.0.1:48982)
0929 10:23:45.025918 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:45.025931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:45.026026 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:45.026596 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:45.026599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:45.026615 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:45.026697 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.026702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.027956 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.027959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.029174 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.029183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.030389 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.030392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.030628 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.030653 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 10:23:45.030727 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 10:23:45.030791 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":50,"thread_start_us":115,"threads_started":1}
I20260929 10:23:45.031607 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.025543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.21.33.131:46599)
0929 10:23:45.025904 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.025916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.026000 (+ 84us) server_negotiation.cc:407] Connection header received
0929 10:23:45.026136 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.026141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.026332 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.026472 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:45.026821 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.027860 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.029328 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.030280 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.030371 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.030494 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.031385 (+ 891us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.031481 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260929 10:23:45.031831 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.025335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.21.33.129:35209)
0929 10:23:45.025694 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.025707 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.025730 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:23:45.025807 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.025813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.026026 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.026224 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:45.026410 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.027860 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.028959 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.030024 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.030130 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.030270 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.031110 (+ 840us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.031207 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":60,"threads_started":1}
I20260929 10:23:45.032680 28800 tablet_service.cc:1467] Processing CreateTablet for tablet f7930c730a864288b404ef414a1f8f2a (DEFAULT_TABLE table=test-table [id=2011358ce8f547f4b25a5acd0ab1c1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:45.033941 28800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7930c730a864288b404ef414a1f8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:45.036021 28663 tablet_service.cc:1467] Processing CreateTablet for tablet f7930c730a864288b404ef414a1f8f2a (DEFAULT_TABLE table=test-table [id=2011358ce8f547f4b25a5acd0ab1c1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:45.036021 28938 tablet_service.cc:1467] Processing CreateTablet for tablet f7930c730a864288b404ef414a1f8f2a (DEFAULT_TABLE table=test-table [id=2011358ce8f547f4b25a5acd0ab1c1ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:45.037060 28938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7930c730a864288b404ef414a1f8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:45.037060 28663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7930c730a864288b404ef414a1f8f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:45.041023 29038 tablet_bootstrap.cc:492] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Bootstrap starting.
I20260929 10:23:45.043087 29038 tablet_bootstrap.cc:654] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:45.043828 29038 log.cc:824] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:45.045845 29040 tablet_bootstrap.cc:492] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Bootstrap starting.
I20260929 10:23:45.045956 29038 tablet_bootstrap.cc:492] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: No bootstrap required, opened a new log
I20260929 10:23:45.046137 29038 ts_tablet_manager.cc:1397] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:23:45.048049 29040 tablet_bootstrap.cc:654] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:45.048909 29040 log.cc:824] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:45.044827 29041 tablet_bootstrap.cc:492] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Bootstrap starting.
I20260929 10:23:45.050549 29038 raft_consensus.cc:348] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.050836 29038 raft_consensus.cc:374] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:45.050897 29038 raft_consensus.cc:729] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f177df46dbe4c1fbef83392d6da2f2d, State: Initialized, Role: FOLLOWER
I20260929 10:23:45.051082 29040 tablet_bootstrap.cc:492] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: No bootstrap required, opened a new log
I20260929 10:23:45.051079 29041 tablet_bootstrap.cc:654] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:45.051270 29040 ts_tablet_manager.cc:1397] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 10:23:45.051347 29038 consensus_queue.cc:260] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.051923 29041 log.cc:824] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:45.052354 29038 ts_tablet_manager.cc:1428] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:23:45.052822 28867 heartbeater.cc:503] Master 127.21.33.190:35965 was elected leader, sending a full tablet report...
I20260929 10:23:45.055799 29040 raft_consensus.cc:348] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.056047 29040 raft_consensus.cc:374] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:45.056097 29040 raft_consensus.cc:729] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccdcd9e0d9a437e9678399ebd420b5e, State: Initialized, Role: FOLLOWER
I20260929 10:23:45.056571 29040 consensus_queue.cc:260] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.057670 29040 ts_tablet_manager.cc:1428] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260929 10:23:45.057886 29005 heartbeater.cc:503] Master 127.21.33.190:35965 was elected leader, sending a full tablet report...
I20260929 10:23:45.059206 29041 tablet_bootstrap.cc:492] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: No bootstrap required, opened a new log
I20260929 10:23:45.059417 29041 ts_tablet_manager.cc:1397] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260929 10:23:45.064076 29041 raft_consensus.cc:348] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.064391 29041 raft_consensus.cc:374] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:45.064463 29041 raft_consensus.cc:729] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb6f3e06f364173bc2e0715ddec5754, State: Initialized, Role: FOLLOWER
I20260929 10:23:45.064951 29041 consensus_queue.cc:260] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.066005 29041 ts_tablet_manager.cc:1428] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260929 10:23:45.067407 28730 heartbeater.cc:503] Master 127.21.33.190:35965 was elected leader, sending a full tablet report...
W20260929 10:23:45.223500 29007 tablet.cc:2369] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:45.223965 29045 raft_consensus.cc:482] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:45.224181 29045 raft_consensus.cc:504] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.225342 29045 leader_election.cc:288] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193), 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:45.230211 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.225696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:37845 (local address 127.21.33.130:35193)
0929 10:23:45.225840 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.225861 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.225880 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:23:45.225939 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.225942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.225992 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.226078 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:45.226366 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.227208 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.228864 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.229507 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.229631 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.229683 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.229999 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.230075 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:23:45.230211 29036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.226037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:43207 (local address 127.21.33.129:35209)
0929 10:23:45.226160 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.226167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.226181 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:45.226229 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.226233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.226282 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.226383 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:45.226632 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.227344 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.228864 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.229507 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.229631 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.229683 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.229999 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.230075 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:23:45.230266 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.225943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35209 (local address 127.21.33.131:43207)
0929 10:23:45.226036 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:45.226048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:45.226110 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:45.226404 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:45.226406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:45.226422 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:45.226499 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.226503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.227446 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.227449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.228727 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.228739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.229631 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.229635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.229834 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.229893 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 10:23:45.229970 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:23:45.230094 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":143}
I20260929 10:23:45.230266 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.225599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:35193 (local address 127.21.33.131:37845)
0929 10:23:45.225729 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:45.225746 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:45.225849 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:45.226099 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:45.226101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:45.226114 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:45.226254 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.226259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.227326 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.227329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.228745 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:45.228752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.229631 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.229635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:45.229789 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.229889 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 10:23:45.229970 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:23:45.230093 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":142}
I20260929 10:23:45.231050 28683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7930c730a864288b404ef414a1f8f2a" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb6f3e06f364173bc2e0715ddec5754" is_pre_election: true
I20260929 10:23:45.231209 28820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7930c730a864288b404ef414a1f8f2a" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" is_pre_election: true
I20260929 10:23:45.231396 28683 raft_consensus.cc:2454] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 0.
I20260929 10:23:45.231519 28820 raft_consensus.cc:2454] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 0.
I20260929 10:23:45.231920 28891 leader_election.cc:302] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb6f3e06f364173bc2e0715ddec5754, 5ccdcd9e0d9a437e9678399ebd420b5e; no voters:
I20260929 10:23:45.232177 29045 raft_consensus.cc:2781] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:45.232268 29045 raft_consensus.cc:482] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:45.232313 29045 raft_consensus.cc:3037] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:45.233563 29045 raft_consensus.cc:504] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.233920 29045 leader_election.cc:288] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 election: Requested vote from peers 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193), 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:45.234368 28683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7930c730a864288b404ef414a1f8f2a" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb6f3e06f364173bc2e0715ddec5754"
I20260929 10:23:45.234366 28820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7930c730a864288b404ef414a1f8f2a" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f177df46dbe4c1fbef83392d6da2f2d"
I20260929 10:23:45.234514 28683 raft_consensus.cc:3037] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:45.234510 28820 raft_consensus.cc:3037] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:45.235561 28820 raft_consensus.cc:2454] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 1.
I20260929 10:23:45.235580 28683 raft_consensus.cc:2454] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 1.
I20260929 10:23:45.235836 28891 leader_election.cc:302] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb6f3e06f364173bc2e0715ddec5754, 5ccdcd9e0d9a437e9678399ebd420b5e; no voters:
I20260929 10:23:45.235994 29045 raft_consensus.cc:2781] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:45.236212 29045 raft_consensus.cc:686] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 LEADER]: Becoming Leader. State: Replica: 5ccdcd9e0d9a437e9678399ebd420b5e, State: Running, Role: LEADER
I20260929 10:23:45.236763 29045 consensus_queue.cc:237] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } }
I20260929 10:23:45.240165 27766 catalog_manager.cc:5495] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131). New cstate: current_term: 1 leader_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:45.266232 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.262724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.21.33.131:46599)
0929 10:23:45.262875 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.262882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.262898 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:45.262988 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.262991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.263041 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.263152 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:45.263399 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.263904 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.264752 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.264973 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.265084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.265411 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:45.265829 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:45.265966 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.266026 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.266082 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 10:23:45.270594 28869 tablet.cc:2369] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:45.271929 28683 raft_consensus.cc:1261] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 1 FOLLOWER]: Refusing update from remote peer 5ccdcd9e0d9a437e9678399ebd420b5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:45.271935 28820 raft_consensus.cc:1261] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Refusing update from remote peer 5ccdcd9e0d9a437e9678399ebd420b5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:45.272629 29045 consensus_queue.cc:1035] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:45.272810 29047 consensus_queue.cc:1035] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:45.284435 29051 mvcc.cc:205] Tried to move back new op lower bound from 7334614733906300928 to 7334614733772640256. Current Snapshot: MvccSnapshot[applied={T|T < 7334614733906300928 or (T in {7334614733906300928})}]
I20260929 10:23:45.285446 29052 mvcc.cc:205] Tried to move back new op lower bound from 7334614733906300928 to 7334614733772640256. Current Snapshot: MvccSnapshot[applied={T|T < 7334614733906300928}]
I20260929 10:23:45.293613 27766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:45.293797 27766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:45.299372 27766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d47f8e27fada4f2d8fb7199955d5eb1c: Sending DeleteTablet for 3 replicas of tablet f7930c730a864288b404ef414a1f8f2a
I20260929 10:23:45.300266 28800 tablet_service.cc:1514] Processing DeleteTablet for tablet f7930c730a864288b404ef414a1f8f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:45 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:51504
I20260929 10:23:45.300428 28938 tablet_service.cc:1514] Processing DeleteTablet for tablet f7930c730a864288b404ef414a1f8f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:45 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:48982
I20260929 10:23:45.300426 28663 tablet_service.cc:1514] Processing DeleteTablet for tablet f7930c730a864288b404ef414a1f8f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:45 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:58220
I20260929 10:23:45.301731 29059 tablet_replica.cc:330] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: stopping tablet replica
I20260929 10:23:45.301976 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 27734
I20260929 10:23:45.310792 29061 tablet_replica.cc:330] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: stopping tablet replica
I20260929 10:23:45.311213 29061 raft_consensus.cc:2229] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:45.311689 29061 raft_consensus.cc:2258] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:45.312108 29060 tablet_replica.cc:330] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: stopping tablet replica
I20260929 10:23:45.312392 29060 raft_consensus.cc:2229] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:45.312733 29060 raft_consensus.cc:2258] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:45.313155 29059 raft_consensus.cc:2229] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:45.313477 29059 raft_consensus.cc:2258] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:45.313949 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.301180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.21.33.189:45771)
0929 10:23:45.301313 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.301318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.301835 (+ 517us) server_negotiation.cc:407] Connection header received
0929 10:23:45.301881 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.301885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.301936 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.302029 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:45.303473 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.304363 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.305555 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.305929 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.306005 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.308134 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:45.308141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:45.308148 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:45.308181 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:45.310134 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.312442 (+ 2308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.312446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.312448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.312541 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.312934 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.312939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.312941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.313176 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:45.313329 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.313652 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.313793 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:23:45.315570 29060 ts_tablet_manager.cc:1905] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:45.315824 29059 ts_tablet_manager.cc:1905] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:45.316465 29061 ts_tablet_manager.cc:1905] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:45.319150 29060 ts_tablet_manager.cc:1918] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:45.319304 29060 log.cc:1196] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/f7930c730a864288b404ef414a1f8f2a
I20260929 10:23:45.319718 29060 ts_tablet_manager.cc:1939] T f7930c730a864288b404ef414a1f8f2a P 2cb6f3e06f364173bc2e0715ddec5754: Deleting consensus metadata
I20260929 10:23:45.319821 29059 ts_tablet_manager.cc:1918] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:45.319913 29059 log.cc:1196] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/f7930c730a864288b404ef414a1f8f2a
I20260929 10:23:45.320219 29059 ts_tablet_manager.cc:1939] T f7930c730a864288b404ef414a1f8f2a P 0f177df46dbe4c1fbef83392d6da2f2d: Deleting consensus metadata
W20260929 10:23:45.321113 28618 connection.cc:441] server connection from 127.0.0.1:58220 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58220 (request call id 1) could send its response
W20260929 10:23:45.321821 28755 connection.cc:441] server connection from 127.0.0.1:51504 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51504 (request call id 1) could send its response
I20260929 10:23:45.323176 29061 ts_tablet_manager.cc:1918] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:45.323315 29061 log.cc:1196] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/f7930c730a864288b404ef414a1f8f2a
I20260929 10:23:45.323608 29061 ts_tablet_manager.cc:1939] T f7930c730a864288b404ef414a1f8f2a P 5ccdcd9e0d9a437e9678399ebd420b5e: Deleting consensus metadata
I20260929 10:23:45.323701 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.301497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.21.33.188:34991)
0929 10:23:45.301628 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.301632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.314518 (+ 12886us) server_negotiation.cc:407] Connection header received
0929 10:23:45.314875 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.314879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.314930 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.314980 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:45.315826 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.316605 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.317749 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.318090 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.318166 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.319908 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:45.319912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:45.319915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:45.319940 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:45.321689 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.322386 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.322390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.322392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.322473 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.322765 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.322769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.322771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.323013 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:45.323161 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.323420 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.323544 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 10:23:45.324837 28891 connection.cc:441] server connection from 127.0.0.1:48982 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48982 (request call id 1) could send its response
I20260929 10:23:45.333917 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.317895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.21.33.188:34991)
0929 10:23:45.318000 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.318004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.325782 (+ 7778us) server_negotiation.cc:407] Connection header received
0929 10:23:45.325896 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.325899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.325947 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.326038 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:45.326855 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.327369 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.328141 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.328392 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.328498 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.330277 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:45.330280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:45.330283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:45.330306 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:45.331887 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.332385 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.332389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.332391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.332473 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.332813 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.332816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.332818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.333079 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:45.333244 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.333504 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.333753 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:23:45.334697 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:45.318181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.21.33.189:45771)
0929 10:23:45.318290 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:45.318294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:45.326130 (+ 7836us) server_negotiation.cc:407] Connection header received
0929 10:23:45.326183 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:45.326189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:45.326237 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:45.326326 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:45.327114 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.327894 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.329116 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:45.329549 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:45.329678 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:45.331130 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:45.331136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:45.331140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:45.331168 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:45.332766 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.333311 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.333315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.333317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.333397 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:45.333689 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:45.333693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:45.333697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:45.333925 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:45.334021 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:45.334400 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 10:23:45.334517 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:46.287068 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.285848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:58001)
0929 10:23:46.286840 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.286940 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":80,"threads_started":1}
I20260929 10:23:46.287364 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.286605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:36575)
0929 10:23:46.287151 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.287265 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":83,"threads_started":1}
W20260929 10:23:46.287417 28867 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:35965 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
W20260929 10:23:46.287774 29005 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:35965 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:46.288117 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.287708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:56045)
0929 10:23:46.287955 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.288032 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
W20260929 10:23:46.288395 28730 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:35965 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:46.799109 29073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d47f8e27fada4f2d8fb7199955d5eb1c)
I20260929 10:23:46.799286 29074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d47f8e27fada4f2d8fb7199955d5eb1c)
I20260929 10:23:46.799362 29073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:46.799479 29074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:46.800377 29074 leader_election.cc:288] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965), 91a4661763b44194adbc81a50a6aea93 (127.21.33.188:34991)
I20260929 10:23:46.800470 29073 leader_election.cc:288] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965), 0b76f60281ac4622af2631809c3d238c (127.21.33.189:45771)
I20260929 10:23:46.801147 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.800713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55614)
0929 10:23:46.800964 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.801047 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":50,"threads_started":1}
I20260929 10:23:46.801195 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.800589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55600)
0929 10:23:46.800997 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.801079 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":73,"threads_started":1}
W20260929 10:23:46.801343 28535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
W20260929 10:23:46.801379 28351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:46.802137 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.801887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55626)
0929 10:23:46.802018 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.802073 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 10:23:46.802294 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.802033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55632)
0929 10:23:46.802178 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.802225 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 10:23:46.802287 28535 leader_election.cc:334] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965): Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
W20260929 10:23:46.802438 28351 leader_election.cc:334] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965): Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:46.805125 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.800949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:34991 (local address 127.0.0.1:55724)
0929 10:23:46.801387 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.801400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:46.801488 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:46.802103 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:46.802106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:46.802127 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:46.802249 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:46.802256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.803203 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.803207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:46.804038 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:46.804046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.804706 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.804709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:46.804801 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:46.804831 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 10:23:46.804919 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 10:23:46.804997 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":50,"threads_started":1}
I20260929 10:23:46.805125 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.801612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.21.33.188:34991)
0929 10:23:46.801877 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:46.801885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:46.801897 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:46.801940 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:46.801942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:46.801987 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:46.802074 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:46.802437 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.803092 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.804174 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.804585 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.804703 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:46.804772 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:46.804944 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 10:23:46.805017 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
I20260929 10:23:46.805361 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.800683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:45771 (local address 127.0.0.1:48872)
0929 10:23:46.800913 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.800929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:46.801037 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:46.802554 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:46.802557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:46.802574 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:46.802648 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:46.802653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.803387 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.803391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:46.804167 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:46.804176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.804826 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.804830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:46.804918 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:46.804947 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 10:23:46.805036 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:23:46.805111 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260929 10:23:46.805676 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.801113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48872 (local address 127.21.33.189:45771)
0929 10:23:46.801726 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:46.801732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:46.801759 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:23:46.801805 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:46.801808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:46.801852 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:46.802601 (+ 749us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:46.802751 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.803270 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.804298 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:46.804705 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:46.804826 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:46.804880 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:46.805051 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 10:23:46.805126 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":548,"thread_start_us":90,"threads_started":1}
I20260929 10:23:46.805637 28560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b76f60281ac4622af2631809c3d238c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91a4661763b44194adbc81a50a6aea93" is_pre_election: true
I20260929 10:23:46.805940 28560 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b76f60281ac4622af2631809c3d238c in term 1.
I20260929 10:23:46.806157 28368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91a4661763b44194adbc81a50a6aea93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b76f60281ac4622af2631809c3d238c" is_pre_election: true
I20260929 10:23:46.806373 28351 leader_election.cc:302] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b76f60281ac4622af2631809c3d238c, 91a4661763b44194adbc81a50a6aea93; no voters: d47f8e27fada4f2d8fb7199955d5eb1c
I20260929 10:23:46.806388 28368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a4661763b44194adbc81a50a6aea93 in term 1.
I20260929 10:23:46.806639 29074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 10:23:46.806700 29074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Starting leader election (detected failure of leader d47f8e27fada4f2d8fb7199955d5eb1c)
I20260929 10:23:46.806730 29074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:23:46.806964 28536 leader_election.cc:302] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b76f60281ac4622af2631809c3d238c, 91a4661763b44194adbc81a50a6aea93; no voters: d47f8e27fada4f2d8fb7199955d5eb1c
I20260929 10:23:46.807183 29073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 10:23:46.807246 29073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d47f8e27fada4f2d8fb7199955d5eb1c)
I20260929 10:23:46.807283 29073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:23:46.808139 29074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:46.808580 29074 leader_election.cc:288] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [CANDIDATE]: Term 2 election: Requested vote from peers d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965), 91a4661763b44194adbc81a50a6aea93 (127.21.33.188:34991)
I20260929 10:23:46.808530 29073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:46.808923 29073 leader_election.cc:288] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 2 election: Requested vote from peers d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965), 0b76f60281ac4622af2631809c3d238c (127.21.33.189:45771)
I20260929 10:23:46.808981 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.808718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55648)
0929 10:23:46.808842 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.808905 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 10:23:46.809163 28560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b76f60281ac4622af2631809c3d238c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91a4661763b44194adbc81a50a6aea93"
I20260929 10:23:46.809284 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.809019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55656)
0929 10:23:46.809114 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.809176 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 10:23:46.809376 28560 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b76f60281ac4622af2631809c3d238c in current term 2: Already voted for candidate 91a4661763b44194adbc81a50a6aea93 in this term.
I20260929 10:23:46.809525 28368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91a4661763b44194adbc81a50a6aea93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b76f60281ac4622af2631809c3d238c"
I20260929 10:23:46.809692 28368 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 91a4661763b44194adbc81a50a6aea93 in current term 2: Already voted for candidate 0b76f60281ac4622af2631809c3d238c in this term.
I20260929 10:23:46.809950 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.809742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55666)
0929 10:23:46.809851 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.809905 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 10:23:46.810039 28351 leader_election.cc:334] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965): Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:46.810102 28351 leader_election.cc:302] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b76f60281ac4622af2631809c3d238c; no voters: 91a4661763b44194adbc81a50a6aea93, d47f8e27fada4f2d8fb7199955d5eb1c
I20260929 10:23:46.810225 29074 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 10:23:46.810295 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:46.810043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55682)
0929 10:23:46.810151 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:46.810210 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 10:23:46.810390 28535 leader_election.cc:334] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965): Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:46.810457 28535 leader_election.cc:302] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91a4661763b44194adbc81a50a6aea93; no voters: 0b76f60281ac4622af2631809c3d238c, d47f8e27fada4f2d8fb7199955d5eb1c
I20260929 10:23:46.810582 29073 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 10:23:47.288538 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:47.287934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:48755)
0929 10:23:47.288346 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:47.288425 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":85,"threads_started":1}
I20260929 10:23:47.289119 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:47.288501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:43417)
0929 10:23:47.288956 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:47.289016 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":71,"threads_started":1}
I20260929 10:23:47.289341 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:47.288895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:36867)
0929 10:23:47.289140 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:47.289253 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260929 10:23:48.290531 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.289347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:56267)
0929 10:23:48.290310 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.290399 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":108,"threads_started":1}
I20260929 10:23:48.291008 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.290086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:37175)
0929 10:23:48.290819 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.290890 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":65,"threads_started":1}
I20260929 10:23:48.291186 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.290520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:33771)
0929 10:23:48.290985 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.291051 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":58,"threads_started":1}
I20260929 10:23:48.654752 29089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:48.654920 29089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:48.655411 29089 leader_election.cc:288] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965), 0b76f60281ac4622af2631809c3d238c (127.21.33.189:45771)
I20260929 10:23:48.655952 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.655552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55966)
0929 10:23:48.655778 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.655853 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":58,"threads_started":1}
I20260929 10:23:48.655933 28368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91a4661763b44194adbc81a50a6aea93" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b76f60281ac4622af2631809c3d238c" is_pre_election: true
I20260929 10:23:48.656083 28368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91a4661763b44194adbc81a50a6aea93 in term 2.
I20260929 10:23:48.656390 28536 leader_election.cc:302] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b76f60281ac4622af2631809c3d238c, 91a4661763b44194adbc81a50a6aea93; no voters:
I20260929 10:23:48.656612 29089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 10:23:48.656674 29089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:48.656728 29089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:23:48.656872 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.656618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55974)
0929 10:23:48.656742 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.656805 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 10:23:48.657032 28535 leader_election.cc:334] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965): Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:48.657868 29089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:48.658195 29089 leader_election.cc:288] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 3 election: Requested vote from peers d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965), 0b76f60281ac4622af2631809c3d238c (127.21.33.189:45771)
I20260929 10:23:48.658519 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.658281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55990)
0929 10:23:48.658374 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.658435 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 10:23:48.658555 28368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91a4661763b44194adbc81a50a6aea93" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b76f60281ac4622af2631809c3d238c"
I20260929 10:23:48.658670 28368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:23:48.659286 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.659075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:55994)
0929 10:23:48.659183 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.659233 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 10:23:48.659420 28535 leader_election.cc:334] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965): Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
I20260929 10:23:48.659819 28368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91a4661763b44194adbc81a50a6aea93 in term 3.
I20260929 10:23:48.660073 28536 leader_election.cc:302] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b76f60281ac4622af2631809c3d238c, 91a4661763b44194adbc81a50a6aea93; no voters: d47f8e27fada4f2d8fb7199955d5eb1c
I20260929 10:23:48.660275 29089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:23:48.660513 29089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [term 3 LEADER]: Becoming Leader. State: Replica: 91a4661763b44194adbc81a50a6aea93, State: Running, Role: LEADER
I20260929 10:23:48.660778 29089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } }
I20260929 10:23:48.662196 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a4661763b44194adbc81a50a6aea93. Latest consensus state: current_term: 3 leader_uuid: "91a4661763b44194adbc81a50a6aea93" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:48.662339 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:48.662638 29094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:48.664846 29094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:48.665180 29094 catalog_manager.cc:1259] Loaded cluster ID: 3d8c5231345e459d85e898a741d21cdf
I20260929 10:23:48.665289 29094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:48.665799 29094 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:48.666129 29094 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Loaded TSK: 0
I20260929 10:23:48.666764 29094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:23:48.712963 28550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:48.716313 28550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:48.719262 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.719006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:56016)
0929 10:23:48.719135 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.719195 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 10:23:48.719419 28368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [term 3 FOLLOWER]: Refusing update from remote peer 91a4661763b44194adbc81a50a6aea93: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 10:23:48.720000 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.719770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.0.0.1:56020)
0929 10:23:48.719883 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.719926 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 10:23:48.719923 29092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 10:23:48.720113 28535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 -> Peer d47f8e27fada4f2d8fb7199955d5eb1c (127.21.33.190:35965): Couldn't send request to peer d47f8e27fada4f2d8fb7199955d5eb1c. Status: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 10:23:48.721369 29096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a4661763b44194adbc81a50a6aea93. Latest consensus state: current_term: 3 leader_uuid: "91a4661763b44194adbc81a50a6aea93" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:48.721494 29096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:23:48.722579 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "91a4661763b44194adbc81a50a6aea93" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:48.722702 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:48.722774 29096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "91a4661763b44194adbc81a50a6aea93" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d47f8e27fada4f2d8fb7199955d5eb1c" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35965 } } peers { permanent_uuid: "0b76f60281ac4622af2631809c3d238c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 45771 } attrs { promote: false } } peers { permanent_uuid: "91a4661763b44194adbc81a50a6aea93" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 34991 } attrs { promote: false } } }
I20260929 10:23:48.722872 29096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b76f60281ac4622af2631809c3d238c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:23:48.733966 29102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.728921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35209 (local address 127.0.0.1:58230)
0929 10:23:48.729246 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.729262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:48.729378 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:48.729924 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:48.729927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:48.729948 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:48.730084 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:48.730094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.731346 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.731349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:48.732550 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:48.732559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.733409 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.733415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:48.733582 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:48.733631 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 10:23:48.733708 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:23:48.733793 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":89,"thread_start_us":154,"threads_started":1}
I20260929 10:23:48.734097 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.729083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.21.33.130:35193)
0929 10:23:48.729942 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:48.729947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:48.729962 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:23:48.730148 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:48.730152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:48.730212 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:48.730584 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:48.730705 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.731464 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.732712 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.733447 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.733623 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:48.733704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:48.733848 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 10:23:48.733930 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":66,"threads_started":1}
I20260929 10:23:48.734143 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.728676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:35193 (local address 127.0.0.1:51512)
0929 10:23:48.728819 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.728837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:48.728937 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:48.730348 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:48.730351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:48.730368 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:48.730468 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:48.730475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.731591 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.731594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:48.732542 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:48.732552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.733622 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.733625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:48.733710 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:48.733731 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:23:48.733833 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:23:48.733917 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":41}
I20260929 10:23:48.734268 29104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.729378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.21.33.129:35209)
0929 10:23:48.729640 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:48.729645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:48.729662 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:23:48.729721 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:48.729724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:48.729782 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:48.730274 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:48.730410 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.731228 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.732669 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.733304 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.733417 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:48.733540 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:48.734014 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 10:23:48.734122 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260929 10:23:48.734990 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.729480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:46599 (local address 127.0.0.1:48990)
0929 10:23:48.729740 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.729754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:48.729845 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:48.730296 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:48.730299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:48.730314 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:48.730401 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:48.730415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.731148 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.731154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:48.732443 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:48.732454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.733438 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.733442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:48.733550 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:48.734692 (+ 1142us) client_negotiation.cc:769] Sending connection context
0929 10:23:48.734791 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 10:23:48.734867 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":1110,"thread_start_us":107,"threads_started":1}
I20260929 10:23:48.735047 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.729595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.21.33.131:46599)
0929 10:23:48.730052 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:48.730059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:48.730075 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:48.730125 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:48.730129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:48.730178 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:48.730290 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:48.730519 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.731038 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.732593 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.733276 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.733610 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:48.733669 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:48.734814 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0929 10:23:48.734921 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":62,"threads_started":1}
I20260929 10:23:48.735008 28663 tablet_service.cc:1467] Processing CreateTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca (DEFAULT_TABLE table=test-table [id=0213e65170b441e9999f93c8d2c8244a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:48.735348 28663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a80ba8d7f864d359a4b481ad6144dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:48.735536 28938 tablet_service.cc:1467] Processing CreateTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca (DEFAULT_TABLE table=test-table [id=0213e65170b441e9999f93c8d2c8244a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:48.735838 28938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a80ba8d7f864d359a4b481ad6144dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:48.737927 29107 tablet_bootstrap.cc:492] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754: Bootstrap starting.
I20260929 10:23:48.738106 29108 tablet_bootstrap.cc:492] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: Bootstrap starting.
I20260929 10:23:48.738204 28800 tablet_service.cc:1467] Processing CreateTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca (DEFAULT_TABLE table=test-table [id=0213e65170b441e9999f93c8d2c8244a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:48.738515 28800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a80ba8d7f864d359a4b481ad6144dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:48.739008 29107 tablet_bootstrap.cc:654] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:48.739095 29108 tablet_bootstrap.cc:654] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:48.740250 29108 tablet_bootstrap.cc:492] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: No bootstrap required, opened a new log
I20260929 10:23:48.740353 29108 ts_tablet_manager.cc:1397] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:48.740703 29108 raft_consensus.cc:348] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.740828 29108 raft_consensus.cc:374] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:48.740875 29108 raft_consensus.cc:729] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccdcd9e0d9a437e9678399ebd420b5e, State: Initialized, Role: FOLLOWER
I20260929 10:23:48.741004 29108 consensus_queue.cc:260] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.741232 29111 tablet_bootstrap.cc:492] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d: Bootstrap starting.
I20260929 10:23:48.741428 29108 ts_tablet_manager.cc:1428] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:48.742183 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.741898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:33189)
0929 10:23:48.742015 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.742085 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 10:23:48.743016 29107 tablet_bootstrap.cc:492] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754: No bootstrap required, opened a new log
I20260929 10:23:48.743125 29107 ts_tablet_manager.cc:1397] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 10:23:48.743155 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.742930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:53973)
0929 10:23:48.743037 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.743085 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:23:48.743637 29107 raft_consensus.cc:348] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.743765 29107 raft_consensus.cc:374] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:48.743803 29107 raft_consensus.cc:729] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb6f3e06f364173bc2e0715ddec5754, State: Initialized, Role: FOLLOWER
I20260929 10:23:48.743976 29107 consensus_queue.cc:260] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.744139 29111 tablet_bootstrap.cc:654] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:48.744133 29004 heartbeater.cc:503] Master 127.21.33.188:34991 was elected leader, sending a full tablet report...
I20260929 10:23:48.744386 29107 ts_tablet_manager.cc:1428] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 10:23:48.745112 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.744841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:42205)
0929 10:23:48.744986 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.745044 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 10:23:48.746222 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.746009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:60875)
0929 10:23:48.746110 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.746161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 10:23:48.746328 28729 heartbeater.cc:503] Master 127.21.33.188:34991 was elected leader, sending a full tablet report...
I20260929 10:23:48.747012 29111 tablet_bootstrap.cc:492] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d: No bootstrap required, opened a new log
I20260929 10:23:48.747124 29111 ts_tablet_manager.cc:1397] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260929 10:23:48.747632 29111 raft_consensus.cc:348] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.747766 29111 raft_consensus.cc:374] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:48.747846 29111 raft_consensus.cc:729] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f177df46dbe4c1fbef83392d6da2f2d, State: Initialized, Role: FOLLOWER
I20260929 10:23:48.747996 29111 consensus_queue.cc:260] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.748354 29111 ts_tablet_manager.cc:1428] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 10:23:48.748899 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.748638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:45417)
0929 10:23:48.748765 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.748825 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 10:23:48.749811 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.749576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:48409)
0929 10:23:48.749685 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.749735 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:23:48.749886 28866 heartbeater.cc:503] Master 127.21.33.188:34991 was elected leader, sending a full tablet report...
W20260929 10:23:48.799124 28732 tablet.cc:2369] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:48.867583 29112 raft_consensus.cc:482] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:48.867756 29112 raft_consensus.cc:504] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.868227 29112 leader_election.cc:288] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193), 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:48.868780 28820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80ba8d7f864d359a4b481ad6144dca" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" is_pre_election: true
I20260929 10:23:48.868816 28683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80ba8d7f864d359a4b481ad6144dca" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb6f3e06f364173bc2e0715ddec5754" is_pre_election: true
I20260929 10:23:48.868954 28820 raft_consensus.cc:2454] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 0.
I20260929 10:23:48.868976 28683 raft_consensus.cc:2454] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 0.
I20260929 10:23:48.869371 28891 leader_election.cc:302] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f177df46dbe4c1fbef83392d6da2f2d, 5ccdcd9e0d9a437e9678399ebd420b5e; no voters:
I20260929 10:23:48.869561 29112 raft_consensus.cc:2781] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:48.869623 29112 raft_consensus.cc:482] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:48.869663 29112 raft_consensus.cc:3037] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:48.870646 29112 raft_consensus.cc:504] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.870990 29112 leader_election.cc:288] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 election: Requested vote from peers 0f177df46dbe4c1fbef83392d6da2f2d (127.21.33.130:35193), 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209)
I20260929 10:23:48.871397 28683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80ba8d7f864d359a4b481ad6144dca" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb6f3e06f364173bc2e0715ddec5754"
I20260929 10:23:48.871419 28820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80ba8d7f864d359a4b481ad6144dca" candidate_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f177df46dbe4c1fbef83392d6da2f2d"
I20260929 10:23:48.871527 28820 raft_consensus.cc:3037] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:48.871528 28683 raft_consensus.cc:3037] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:48.872630 28820 raft_consensus.cc:2454] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 1.
I20260929 10:23:48.872630 28683 raft_consensus.cc:2454] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccdcd9e0d9a437e9678399ebd420b5e in term 1.
I20260929 10:23:48.872938 28891 leader_election.cc:302] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb6f3e06f364173bc2e0715ddec5754, 5ccdcd9e0d9a437e9678399ebd420b5e; no voters:
I20260929 10:23:48.873103 29112 raft_consensus.cc:2781] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:48.873348 29112 raft_consensus.cc:686] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 LEADER]: Becoming Leader. State: Replica: 5ccdcd9e0d9a437e9678399ebd420b5e, State: Running, Role: LEADER
I20260929 10:23:48.873509 29112 consensus_queue.cc:237] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } }
I20260929 10:23:48.874295 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.873846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:40973)
0929 10:23:48.874120 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.874213 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260929 10:23:48.875204 28548 catalog_manager.cc:5495] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131). New cstate: current_term: 1 leader_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccdcd9e0d9a437e9678399ebd420b5e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46599 } health_report { overall_health: HEALTHY } } }
I20260929 10:23:48.922950 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.920177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.21.33.131:46599)
0929 10:23:48.920299 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:48.920303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:48.920318 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:23:48.920363 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:48.920365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:48.920407 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:48.920477 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:48.920716 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.921289 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.922124 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:48.922330 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:48.922403 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:48.922610 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:48.922674 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:48.922725 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:48.922803 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 10:23:48.922845 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:23:48.925344 28820 raft_consensus.cc:1261] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Refusing update from remote peer 5ccdcd9e0d9a437e9678399ebd420b5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:48.925344 28683 raft_consensus.cc:1261] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754 [term 1 FOLLOWER]: Refusing update from remote peer 5ccdcd9e0d9a437e9678399ebd420b5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:48.925825 29116 consensus_queue.cc:1035] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f177df46dbe4c1fbef83392d6da2f2d" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 35193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:48.926035 29112 consensus_queue.cc:1035] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb6f3e06f364173bc2e0715ddec5754" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:48.927809 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.927242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:48571)
0929 10:23:48.927670 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.927731 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260929 10:23:48.928411 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.927972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:34535)
0929 10:23:48.928282 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.928339 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260929 10:23:48.930742 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.930415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:38189)
0929 10:23:48.930617 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.930672 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 10:23:48.931191 29119 mvcc.cc:205] Tried to move back new op lower bound from 7334614748873256960 to 7334614748667371520. Current Snapshot: MvccSnapshot[applied={T|T < 7334614748873256960}]
I20260929 10:23:48.933132 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.932841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.129:35227)
0929 10:23:48.932995 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.933055 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 10:23:48.935153 29087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.934914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.131:51399)
0929 10:23:48.935026 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.935077 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 10:23:48.935724 28550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:48.935904 28550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:48.936785 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.936492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35965 (local address 127.21.33.130:47273)
0929 10:23:48.936658 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.936716 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 10:23:48.940553 28550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91a4661763b44194adbc81a50a6aea93: Sending DeleteTablet for 3 replicas of tablet 3a80ba8d7f864d359a4b481ad6144dca
I20260929 10:23:48.941406 28663 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:48 UTC) from {username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:58230
I20260929 10:23:48.941627 28799 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:48 UTC) from {username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:51512
I20260929 10:23:48.941736 28800 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:48 UTC) from {username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:51512
I20260929 10:23:48.941887 28938 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:48 UTC) from {username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:48990
I20260929 10:23:48.941959 28937 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80ba8d7f864d359a4b481ad6144dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:48 UTC) from {username='oryx', principal='oryx/127.21.33.188@KRBTEST.COM'} at 127.0.0.1:48990
I20260929 10:23:48.942327 29130 tablet_replica.cc:330] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: stopping tablet replica
I20260929 10:23:48.942473 29130 raft_consensus.cc:2229] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:48.942514 29128 tablet_replica.cc:330] T 3a80ba8d7f864d359a4b481ad6144dca P 2cb6f3e06f364173bc2e0715ddec5754: stopping tablet replica
I20260929 10:23:48.942665 29130 raft_consensus.cc:2258] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:48.942677 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 28511
I20260929 10:23:48.943742 29129 tablet_replica.cc:330] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d: stopping tablet replica
I20260929 10:23:48.943882 29129 raft_consensus.cc:2229] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:48.943897 29130 ts_tablet_manager.cc:1905] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:48.944011 29129 raft_consensus.cc:2258] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:48.947319 29130 ts_tablet_manager.cc:1918] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:48.947413 29130 log.cc:1196] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/3a80ba8d7f864d359a4b481ad6144dca
I20260929 10:23:48.947719 29130 ts_tablet_manager.cc:1939] T 3a80ba8d7f864d359a4b481ad6144dca P 5ccdcd9e0d9a437e9678399ebd420b5e: Deleting consensus metadata
I20260929 10:23:48.948526 28536 catalog_manager.cc:4841] TS 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131:46599): tablet 3a80ba8d7f864d359a4b481ad6144dca (table test-table [id=0213e65170b441e9999f93c8d2c8244a]) successfully deleted
W20260929 10:23:48.948930 28536 catalog_manager.cc:4808] TS 5ccdcd9e0d9a437e9678399ebd420b5e (127.21.33.131:46599): delete failed for tablet 3a80ba8d7f864d359a4b481ad6144dca because the tablet was not found. No further retry: Not found: Tablet not found: 3a80ba8d7f864d359a4b481ad6144dca
I20260929 10:23:48.949805 29129 ts_tablet_manager.cc:1905] T 3a80ba8d7f864d359a4b481ad6144dca P 0f177df46dbe4c1fbef83392d6da2f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:48.951188 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 28737
I20260929 10:23:48.951215 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.950935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35209 (local address 127.0.0.1:58240)
0929 10:23:48.951068 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.951138 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 10:23:48.953677 28534 catalog_manager.cc:4584] TS 2cb6f3e06f364173bc2e0715ddec5754 (127.21.33.129:35209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a80ba8d7f864d359a4b481ad6144dca: Network error: Client connection negotiation failed: client connection to 127.21.33.129:35209: connect: Connection refused (error 111)
I20260929 10:23:48.958401 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 28873
I20260929 10:23:48.958710 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:48.958421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:35193 (local address 127.0.0.1:51526)
0929 10:23:48.958546 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:48.958611 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.130:35193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 10:23:48.965466 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 28230
I20260929 10:23:48.971513 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 28337
2026-09-29T10:23:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7337 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:49 dist-test-slave-2rbl krb5kdc[29133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:49 dist-test-slave-2rbl krb5kdc[29133](info): set up 2 sockets
Sep 29 10:23:49 dist-test-slave-2rbl krb5kdc[29133](info): commencing operation
krb5kdc: starting...
W20260929 10:23:50.018278 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab.
Entry for principal oryx/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab.
I20260929 10:23:50.146659 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:34155
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab
--principal=oryx/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:34155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:50.256563 29146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:50.256904 29146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:50.256981 29146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:50.261106 29146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:50.261206 29146 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 10:23:50.261260 29146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:50.261307 29146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:50.261344 29146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:50.261379 29146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:50.266697 29146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:34155
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.190.keytab
--principal=oryx/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:50.267759 29146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:50.268782 29146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:50.277514 29151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.277535 29154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.277644 29152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:50.277618 29146 server_base.cc:1034] running on GCE node
I20260929 10:23:50.278378 29146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:50.279405 29146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:50.280562 29146 hybrid_clock.cc:648] HybridClock initialized: now 1790677430280548 us; error 27 us; skew 500 ppm
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:50.283358 29146 init.cc:377] Logged in from keytab as oryx/127.21.33.190@KRBTEST.COM (short username oryx)
I20260929 10:23:50.284137 29146 webserver.cc:460] Webserver started at http://127.21.33.190:33033/ using document root <none> and password file <none>
I20260929 10:23:50.284626 29146 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:50.284680 29146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:50.284845 29146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:50.286788 29146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "c5e3f06eccd4423cb11d14a158a3e834"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "2231a9b7814dad36689b440835001509"
server_key_iv: "55ef0e733ac3209faf798daca0f0a31a"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.287266 29146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "c5e3f06eccd4423cb11d14a158a3e834"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "2231a9b7814dad36689b440835001509"
server_key_iv: "55ef0e733ac3209faf798daca0f0a31a"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.291015 29146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 10:23:50.293406 29161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:50.294637 29146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 10:23:50.294731 29146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "c5e3f06eccd4423cb11d14a158a3e834"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "2231a9b7814dad36689b440835001509"
server_key_iv: "55ef0e733ac3209faf798daca0f0a31a"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.294866 29146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:50.310420 29146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:50.313716 29146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:50.313925 29146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:50.321645 29146 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:34155
I20260929 10:23:50.321662 29223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:34155 every 8 connection(s)
I20260929 10:23:50.322659 29146 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:50.325395 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:50.330673 29224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Bootstrap starting.
I20260929 10:23:50.332907 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 29146
I20260929 10:23:50.333045 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:50.333146 29224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:50.333386 21638 external_mini_cluster.cc:1428] Setting key 081b839dab67871c42b16e221f2a3f23
I20260929 10:23:50.334035 29224 log.cc:824] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:50.335879 29224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: No bootstrap required, opened a new log
I20260929 10:23:50.340233 29224 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3f06eccd4423cb11d14a158a3e834" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34155 } }
I20260929 10:23:50.340519 29224 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:50.340583 29224 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e3f06eccd4423cb11d14a158a3e834, State: Initialized, Role: FOLLOWER
I20260929 10:23:50.341120 29224 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3f06eccd4423cb11d14a158a3e834" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34155 } }
I20260929 10:23:50.341336 29224 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:50.341403 29224 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:50.341517 29224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:50.342731 29224 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3f06eccd4423cb11d14a158a3e834" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34155 } }
I20260929 10:23:50.343115 29224 leader_election.cc:302] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5e3f06eccd4423cb11d14a158a3e834; no voters:
I20260929 10:23:50.343510 29224 leader_election.cc:288] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:50.343600 29229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:50.343818 29229 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [term 1 LEADER]: Becoming Leader. State: Replica: c5e3f06eccd4423cb11d14a158a3e834, State: Running, Role: LEADER
I20260929 10:23:50.344136 29229 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3f06eccd4423cb11d14a158a3e834" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34155 } }
I20260929 10:23:50.344801 29224 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:50.346175 29230 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5e3f06eccd4423cb11d14a158a3e834. Latest consensus state: current_term: 1 leader_uuid: "c5e3f06eccd4423cb11d14a158a3e834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3f06eccd4423cb11d14a158a3e834" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34155 } } }
I20260929 10:23:50.346642 29230 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:50.346504 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5e3f06eccd4423cb11d14a158a3e834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5e3f06eccd4423cb11d14a158a3e834" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34155 } } }
I20260929 10:23:50.346982 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834 [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:50.348322 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.334500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.21.33.190:34155)
0929 10:23:50.334944 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:50.334969 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:50.335004 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:23:50.335611 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:50.335628 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:50.335935 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:50.336250 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:50.337143 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.338145 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.339349 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.339797 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.339909 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.342889 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:50.342896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:50.342918 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:50.342945 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:50.345088 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.345637 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.345644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.345653 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.345806 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.346146 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.346150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.346151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.346662 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:50.346854 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:50.347260 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 10:23:50.347550 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":170,"threads_started":1}
W20260929 10:23:50.350451 29244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:23:50.350553 29244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:23:50.350617 29243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:50.351271 29243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:50.356535 29243 catalog_manager.cc:1347] Generated new cluster ID: b84cabe45bac4d8aa9e3f8b537159e5e
I20260929 10:23:50.356600 29243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:50.376580 29243 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:50.377574 29243 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:50.385656 29243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Generated new TSK 0
I20260929 10:23:50.386365 29243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab.
Entry for principal oryx/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab.
I20260929 10:23:50.424872 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:34155
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab
--principal=oryx/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:50.536572 29250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:50.536868 29250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:50.536945 29250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:50.541020 29250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:50.541116 29250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:50.541208 29250 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:50.541289 29250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:50.546733 29250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab
--principal=oryx/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:34155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:50.547770 29250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:50.548785 29250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:50.558097 29256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.558099 29258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.558079 29255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:50.558391 29250 server_base.cc:1034] running on GCE node
I20260929 10:23:50.558776 29250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:50.559383 29250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:50.560549 29250 hybrid_clock.cc:648] HybridClock initialized: now 1790677430560519 us; error 27 us; skew 500 ppm
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:50.563398 29250 init.cc:377] Logged in from keytab as oryx/127.21.33.129@KRBTEST.COM (short username oryx)
I20260929 10:23:50.564216 29250 webserver.cc:460] Webserver started at http://127.21.33.129:39223/ using document root <none> and password file <none>
I20260929 10:23:50.564761 29250 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:50.564823 29250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:50.565065 29250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:50.567067 29250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "b2b327f9f45d42f19737f9ae6e3ca807"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "b93fee497ef89d80313f2cc344912f79"
server_key_iv: "e52ab4ae17f96a0fdb1994b9060a34dd"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.567620 29250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "b2b327f9f45d42f19737f9ae6e3ca807"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "b93fee497ef89d80313f2cc344912f79"
server_key_iv: "e52ab4ae17f96a0fdb1994b9060a34dd"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.571292 29250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:50.573660 29265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:50.574834 29250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:50.574965 29250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "b2b327f9f45d42f19737f9ae6e3ca807"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "b93fee497ef89d80313f2cc344912f79"
server_key_iv: "e52ab4ae17f96a0fdb1994b9060a34dd"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.575068 29250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:50.591187 29250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:50.594108 29250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:50.594312 29250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:50.595736 29250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:50.595803 29250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:50.595876 29250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:50.595919 29250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:50.606815 29250 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:35021
I20260929 10:23:50.606874 29380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:35021 every 8 connection(s)
I20260929 10:23:50.607895 29250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:50.609490 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.596564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:51595 (local address 127.21.33.190:34155)
0929 10:23:50.596732 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:50.596740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:50.597573 (+ 833us) server_negotiation.cc:407] Connection header received
0929 10:23:50.598583 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:50.598587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:50.598645 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:50.598727 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:50.600054 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.600976 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.602230 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.602633 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.602763 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.605633 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:50.605637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:50.605640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:50.605669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:50.607330 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.607832 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.607838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.607840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.607936 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.608345 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.608350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.608352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.608583 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:50.608685 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:50.609241 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 10:23:50.609369 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:23:50.610285 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.596943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34155 (local address 127.21.33.129:51595)
0929 10:23:50.597415 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:50.597464 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:50.598365 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:50.599073 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:50.599086 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:50.599438 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:50.599896 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.599910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.601135 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.601139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.602123 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.602132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.602833 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.602839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.603085 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.603710 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:50.603728 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:50.605443 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:50.607440 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.607444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.607455 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.607699 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.608060 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.608065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.608072 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.608222 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.608678 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:50.608683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:50.608963 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 10:23:50.609126 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 10:23:50.609452 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":138,"threads_started":1}
I20260929 10:23:50.611631 29381 heartbeater.cc:347] Connected to a master server at 127.21.33.190:34155
I20260929 10:23:50.611677 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 29250
I20260929 10:23:50.611797 29381 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:50.611810 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:50.612107 21638 external_mini_cluster.cc:1428] Setting key 9315c46354d2b7aa1b1506e96ebb0553
I20260929 10:23:50.612361 29381 heartbeater.cc:511] Master 127.21.33.190:34155 requested a full tablet report, sending...
I20260929 10:23:50.613924 29178 ts_manager.cc:194] Registered new tserver with Master: b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021)
I20260929 10:23:50.615015 29178 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:51595
WARNING: no policy specified for oryx/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:23:50.623368 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.613946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:55565 (local address 127.21.33.190:34155)
0929 10:23:50.614076 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:50.614080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:50.615879 (+ 1799us) server_negotiation.cc:407] Connection header received
0929 10:23:50.615998 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:50.616001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:50.616049 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:50.616131 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:50.616707 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.617405 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.618766 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.618976 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.619053 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.620394 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:50.620397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:50.620399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:50.620419 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:50.621831 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.622329 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.622331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.622333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.622402 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.622646 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.622648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.622649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.622805 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:50.622873 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:50.623066 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 10:23:50.623155 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:23:50.623373 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.615431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34155 (local address 127.21.33.129:55565)
0929 10:23:50.615781 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:50.615797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:50.615904 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:50.616168 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:50.616172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:50.616418 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:50.616575 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.616586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.617531 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.617537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.618647 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.618662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.619094 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.619101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.619232 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.619767 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:50.619787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:50.620261 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:50.621936 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.621939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.621941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.622219 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.622496 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.622499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.622500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.622577 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.622880 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:50.622882 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:50.622955 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 10:23:50.623035 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:23:50.623197 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab.
Entry for principal oryx/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab.
I20260929 10:23:50.639673 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:34155
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab
--principal=oryx/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:50.748385 29390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:50.748653 29390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:50.748699 29390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:50.752648 29390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:50.752707 29390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:50.752746 29390 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:50.752784 29390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:50.758157 29390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab
--principal=oryx/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:34155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:50.759163 29390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:50.760108 29390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:50.769345 29395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.769345 29398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.769419 29396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:50.769440 29390 server_base.cc:1034] running on GCE node
I20260929 10:23:50.770076 29390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:50.770615 29390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:50.771780 29390 hybrid_clock.cc:648] HybridClock initialized: now 1790677430771761 us; error 25 us; skew 500 ppm
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:50.774515 29390 init.cc:377] Logged in from keytab as oryx/127.21.33.130@KRBTEST.COM (short username oryx)
I20260929 10:23:50.775382 29390 webserver.cc:460] Webserver started at http://127.21.33.130:42837/ using document root <none> and password file <none>
I20260929 10:23:50.775915 29390 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:50.775974 29390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:50.776221 29390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:50.778286 29390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "992dcee63bc743bb98db0cdb48aace54"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "ebc0b42c028dea3c3de44ac28b37040e"
server_key_iv: "5a26d84e42babe5bf85e3bc8c0ec30bf"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.778921 29390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "992dcee63bc743bb98db0cdb48aace54"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "ebc0b42c028dea3c3de44ac28b37040e"
server_key_iv: "5a26d84e42babe5bf85e3bc8c0ec30bf"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.782529 29390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:50.784912 29405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:50.786214 29390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:50.786337 29390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "992dcee63bc743bb98db0cdb48aace54"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "ebc0b42c028dea3c3de44ac28b37040e"
server_key_iv: "5a26d84e42babe5bf85e3bc8c0ec30bf"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.786439 29390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:50.804702 29390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:50.808025 29390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:50.808230 29390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:50.809787 29390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:50.809863 29390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:50.809934 29390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:50.809978 29390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:50.820864 29390 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:36499
I20260929 10:23:50.821287 29520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:36499 every 8 connection(s)
I20260929 10:23:50.821899 29390 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:50.823963 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.810531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:43287 (local address 127.21.33.190:34155)
0929 10:23:50.810701 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:50.810707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:50.811613 (+ 906us) server_negotiation.cc:407] Connection header received
0929 10:23:50.813005 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:50.813009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:50.813079 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:50.813190 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:50.814835 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.815701 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.817154 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.817445 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.817562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.820341 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:50.820345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:50.820348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:50.820380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:50.821919 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.822516 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.822520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.822522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.822622 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.823013 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.823018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.823021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.823223 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:50.823306 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:50.823735 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 10:23:50.823848 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 10:23:50.824828 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.810957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34155 (local address 127.21.33.130:43287)
0929 10:23:50.811438 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:50.811496 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:50.812711 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:50.813431 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:50.813443 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:50.813925 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:50.814692 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.814715 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.815882 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.815887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.817034 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.817047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.817612 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.817616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.817791 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.818435 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:50.818451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:50.820155 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:50.822040 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.822046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.822059 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.822401 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.822731 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.822742 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.822744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.822913 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.823313 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:50.823318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:50.823534 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 10:23:50.823687 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 10:23:50.823947 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260929 10:23:50.826084 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 29390
I20260929 10:23:50.826134 29521 heartbeater.cc:347] Connected to a master server at 127.21.33.190:34155
I20260929 10:23:50.826213 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:50.826294 29521 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:50.826558 21638 external_mini_cluster.cc:1428] Setting key c1ea9e0628a7c01617ce60e8a11d2e24
I20260929 10:23:50.826809 29521 heartbeater.cc:511] Master 127.21.33.190:34155 requested a full tablet report, sending...
I20260929 10:23:50.827880 29178 ts_manager.cc:194] Registered new tserver with Master: 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499)
I20260929 10:23:50.828630 29178 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:43287
I20260929 10:23:50.836771 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.828452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34155 (local address 127.21.33.130:42421)
0929 10:23:50.828709 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:50.828724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:50.828810 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:50.829714 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:50.829717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:50.829942 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:50.830053 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.830060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.831146 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.831148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.831682 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:50.831689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.832330 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.832335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:50.832441 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.832968 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:50.832978 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:50.833380 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:50.835313 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.835316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.835317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.835502 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.835804 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:50.835806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:50.835808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:50.835879 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:50.836368 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:50.836370 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:50.836440 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 10:23:50.836517 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:23:50.836620 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260929 10:23:50.836895 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:50.829324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:42421 (local address 127.21.33.190:34155)
0929 10:23:50.829473 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:50.829480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:50.829496 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:23:50.829538 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:50.829543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:50.829595 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:50.829708 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:50.830177 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.831011 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.831799 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:50.832199 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:50.832323 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:50.833516 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:50.833523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:50.833527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:50.833559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:50.835208 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.835608 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.835614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.835617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.835707 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:50.835965 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:50.835968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:50.835970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:50.836272 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:50.836390 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:50.836624 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 10:23:50.836745 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab.
Entry for principal oryx/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab.
I20260929 10:23:50.855955 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:34155
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab
--principal=oryx/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:50.964716 29530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:50.964977 29530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:50.965063 29530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:50.969089 29530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:50.969151 29530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:50.969235 29530 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 10:23:50.969303 29530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:50.974491 29530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab
--principal=oryx/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.33.190:34155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:50.975483 29530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:50.976473 29530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:50.985917 29538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.985970 29535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:50.985917 29536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:50.986183 29530 server_base.cc:1034] running on GCE node
I20260929 10:23:50.986579 29530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:50.987176 29530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:50.988332 29530 hybrid_clock.cc:648] HybridClock initialized: now 1790677430988317 us; error 25 us; skew 500 ppm
Sep 29 10:23:50 dist-test-slave-2rbl krb5kdc[29133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:50.991178 29530 init.cc:377] Logged in from keytab as oryx/127.21.33.131@KRBTEST.COM (short username oryx)
I20260929 10:23:50.992043 29530 webserver.cc:460] Webserver started at http://127.21.33.131:36057/ using document root <none> and password file <none>
I20260929 10:23:50.992591 29530 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:50.992679 29530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:50.992910 29530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:50.994913 29530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "448c674331194ea8afa98a5290d97334"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "93b40d1f76090db2b54b252f53564f9a"
server_key_iv: "af7391988161a8c7fef7dbc25ac9b513"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.995455 29530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "448c674331194ea8afa98a5290d97334"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "93b40d1f76090db2b54b252f53564f9a"
server_key_iv: "af7391988161a8c7fef7dbc25ac9b513"
server_key_version: "encryptionkey@0"
I20260929 10:23:50.999192 29530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:51.001708 29545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:51.003002 29530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:51.003132 29530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "448c674331194ea8afa98a5290d97334"
format_stamp: "Formatted at 2026-09-29 10:23:50 on dist-test-slave-2rbl"
server_key: "93b40d1f76090db2b54b252f53564f9a"
server_key_iv: "af7391988161a8c7fef7dbc25ac9b513"
server_key_version: "encryptionkey@0"
I20260929 10:23:51.003239 29530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:51.025880 29530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:51.028906 29530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:51.029120 29530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:51.030539 29530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:51.030613 29530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:51.030685 29530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:51.030740 29530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:51.041074 29530 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:38549
I20260929 10:23:51.041145 29660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:38549 every 8 connection(s)
I20260929 10:23:51.042076 29530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:51.044492 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.031445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:55931 (local address 127.21.33.190:34155)
0929 10:23:51.031637 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.031642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.032476 (+ 834us) server_negotiation.cc:407] Connection header received
0929 10:23:51.033566 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.033573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.033653 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.033717 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:51.035483 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.036385 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.037899 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.038138 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.038252 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.040713 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:51.040717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:51.040720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:51.040752 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:51.042460 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.043059 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.043062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.043064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.043143 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.043496 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.043499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.043500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.043671 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:51.043746 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.044202 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.044358 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:23:51.045472 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.031879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34155 (local address 127.21.33.131:55931)
0929 10:23:51.032314 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.032363 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.033318 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.034193 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.034201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.034566 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:51.035298 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.035315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.036530 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.036534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.037768 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.037777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.038302 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.038319 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.038498 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.039093 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:51.039113 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:51.040541 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:51.042609 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:51.042615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:51.042629 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:51.042944 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:51.043253 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:51.043255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:51.043256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:51.043408 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:51.043768 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:51.043773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:51.043992 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.044159 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.044497 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":138,"threads_started":1}
I20260929 10:23:51.046712 29661 heartbeater.cc:347] Connected to a master server at 127.21.33.190:34155
I20260929 10:23:51.046851 29661 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:51.047341 29661 heartbeater.cc:511] Master 127.21.33.190:34155 requested a full tablet report, sending...
I20260929 10:23:51.048309 29177 ts_manager.cc:194] Registered new tserver with Master: 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:51.049072 29177 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:55931
I20260929 10:23:51.052208 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 29530
I20260929 10:23:51.052351 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:51.052654 21638 external_mini_cluster.cc:1428] Setting key b99e27355c2327989f610f05797c65b0
I20260929 10:23:51.054824 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:23:51.056013 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.048864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34155 (local address 127.21.33.131:52485)
0929 10:23:51.049136 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.049148 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.049296 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.049549 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.049552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.049750 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:51.049874 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.049879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.050605 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.050608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.051171 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.051178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.051630 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.051634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.051735 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.052242 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:51.052254 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:51.052652 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:51.054629 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:51.054632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:51.054634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:51.054825 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:51.055134 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:51.055136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:51.055137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:51.055208 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:51.055642 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:51.055644 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:51.055714 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.055790 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.055888 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":6,"thread_start_us":114,"threads_started":1}
I20260929 10:23:51.056229 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.048934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:52485 (local address 127.21.33.190:34155)
0929 10:23:51.049057 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.049061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.049267 (+ 206us) server_negotiation.cc:407] Connection header received
0929 10:23:51.049400 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.049403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.049445 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.049534 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:51.049983 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.050483 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.051286 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.051523 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.051630 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.052787 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:51.052791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:51.052793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:51.052817 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:51.054517 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.054955 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.054958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.054960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.055045 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.055295 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.055299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.055302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.055534 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:51.055648 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.055872 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.056004 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:23:51.079188 29228 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:51.072962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.21.33.190:34155)
0929 10:23:51.073103 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.073106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.073119 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:51.073158 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.073160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.073247 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.073337 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:51.074027 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.074654 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.075491 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.075711 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.075819 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.079032 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34872: BlockingRecv error: recv got EOF from 127.0.0.1:34872 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:51.087905 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.080520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.21.33.190:34155)
0929 10:23:51.080637 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.080640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.080705 (+ 65us) server_negotiation.cc:407] Connection header received
0929 10:23:51.080796 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.080798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.080832 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.080896 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:51.081495 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.081958 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.082717 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.082936 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.083024 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.085136 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:51.085139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:51.085141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:51.085164 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:51.086569 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.086919 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.086921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.086923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.086990 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.087279 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.087281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.087282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.087429 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:51.087526 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.087708 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.087798 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:51.090869 29177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:51.093786 29177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:51.107568 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.101682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35021 (local address 127.0.0.1:53708)
0929 10:23:51.102048 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.102106 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.102235 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.102957 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.102962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.102982 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.103136 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.103143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.104351 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.104354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.105771 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.105781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.107058 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.107060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.107145 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.107174 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.107298 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.107419 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 10:23:51.107577 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.101842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.21.33.129:35021)
0929 10:23:51.102125 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.102141 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.102168 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:23:51.102417 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.102423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.102637 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.102817 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.103362 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.104230 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.105875 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.106673 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.106815 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.107004 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.107331 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.107425 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260929 10:23:51.107874 29685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.102703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:38549 (local address 127.0.0.1:60866)
0929 10:23:51.103138 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.103150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.103220 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.103909 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.103912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.103930 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.104012 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.104018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.105460 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.105463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.106349 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.106357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.107488 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.107491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.107576 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.107599 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.107652 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.107720 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":97,"threads_started":1}
I20260929 10:23:51.108433 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.102406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:36499 (local address 127.0.0.1:34518)
0929 10:23:51.103079 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.103094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.103188 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.104094 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.104098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.104117 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.104212 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.104218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.105265 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.105268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.106479 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.106490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.107661 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.107664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.107752 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.107771 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.108244 (+ 473us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.108317 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260929 10:23:51.108489 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.103129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.21.33.131:38549)
0929 10:23:51.103452 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.103464 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.103487 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:23:51.103559 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.103564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.103768 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.103911 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.104130 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.105329 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.106511 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.107345 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.108056 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.108207 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.108271 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.108359 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260929 10:23:51.108886 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.102724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.21.33.130:36499)
0929 10:23:51.103453 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.103464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.103487 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:23:51.103559 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.103564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.103756 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.103911 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.104343 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.105124 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.106598 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.107365 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.107467 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.108164 (+ 697us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.108639 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.108754 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":88,"threads_started":1}
I20260929 10:23:51.110893 29595 tablet_service.cc:1467] Processing CreateTablet for tablet 91648e54c8834adeb58409210b1d6d9b (DEFAULT_TABLE table=test-table [id=83634a7d9f9d4b76a782bc44d59ea6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.110986 29315 tablet_service.cc:1467] Processing CreateTablet for tablet 91648e54c8834adeb58409210b1d6d9b (DEFAULT_TABLE table=test-table [id=83634a7d9f9d4b76a782bc44d59ea6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.111284 29455 tablet_service.cc:1467] Processing CreateTablet for tablet 91648e54c8834adeb58409210b1d6d9b (DEFAULT_TABLE table=test-table [id=83634a7d9f9d4b76a782bc44d59ea6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.112044 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91648e54c8834adeb58409210b1d6d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.112044 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91648e54c8834adeb58409210b1d6d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.112164 29455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91648e54c8834adeb58409210b1d6d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.118497 29689 tablet_bootstrap.cc:492] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Bootstrap starting.
I20260929 10:23:51.119850 29691 tablet_bootstrap.cc:492] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Bootstrap starting.
I20260929 10:23:51.120703 29689 tablet_bootstrap.cc:654] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.121120 29690 tablet_bootstrap.cc:492] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Bootstrap starting.
I20260929 10:23:51.121661 29689 log.cc:824] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:51.121735 29691 tablet_bootstrap.cc:654] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.122532 29691 log.cc:824] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:51.123605 29690 tablet_bootstrap.cc:654] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.123806 29689 tablet_bootstrap.cc:492] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: No bootstrap required, opened a new log
I20260929 10:23:51.124006 29689 ts_tablet_manager.cc:1397] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:23:51.124303 29691 tablet_bootstrap.cc:492] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: No bootstrap required, opened a new log
I20260929 10:23:51.124372 29690 log.cc:824] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:51.124468 29691 ts_tablet_manager.cc:1397] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:23:51.126718 29690 tablet_bootstrap.cc:492] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: No bootstrap required, opened a new log
I20260929 10:23:51.126914 29690 ts_tablet_manager.cc:1397] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 10:23:51.127803 29689 raft_consensus.cc:348] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.128048 29689 raft_consensus.cc:374] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.128100 29689 raft_consensus.cc:729] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448c674331194ea8afa98a5290d97334, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.128253 29691 raft_consensus.cc:348] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.128537 29691 raft_consensus.cc:374] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.128593 29691 raft_consensus.cc:729] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992dcee63bc743bb98db0cdb48aace54, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.128589 29689 consensus_queue.cc:260] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.129045 29691 consensus_queue.cc:260] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.129314 29661 heartbeater.cc:503] Master 127.21.33.190:34155 was elected leader, sending a full tablet report...
I20260929 10:23:51.129676 29689 ts_tablet_manager.cc:1428] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:23:51.130012 29691 ts_tablet_manager.cc:1428] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:23:51.130620 29521 heartbeater.cc:503] Master 127.21.33.190:34155 was elected leader, sending a full tablet report...
I20260929 10:23:51.130821 29690 raft_consensus.cc:348] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.131040 29690 raft_consensus.cc:374] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.131093 29690 raft_consensus.cc:729] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.131476 29690 consensus_queue.cc:260] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.132014 29381 heartbeater.cc:503] Master 127.21.33.190:34155 was elected leader, sending a full tablet report...
I20260929 10:23:51.132350 29690 ts_tablet_manager.cc:1428] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260929 10:23:51.161880 29695 raft_consensus.cc:482] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:51.162122 29695 raft_consensus.cc:504] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.163300 29695 leader_election.cc:288] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021), 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499)
I20260929 10:23:51.168396 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.163532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35021 (local address 127.21.33.131:42977)
0929 10:23:51.163699 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.163717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.163842 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.164145 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.164147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.164166 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.164324 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.164335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.165468 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.165473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.166976 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.166991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.167881 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.167884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.168050 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.168072 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.168176 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.168255 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 10:23:51.168439 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.163619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:42977 (local address 127.21.33.129:35021)
0929 10:23:51.163761 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.163766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.163781 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:23:51.163976 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.163982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.164037 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.164120 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.164440 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.165332 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.167132 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.167761 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.167880 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.167940 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.168191 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.168310 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:23:51.169294 29698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.163834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:36499 (local address 127.21.33.131:56445)
0929 10:23:51.164174 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.164187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.164276 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.164538 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.164540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.164554 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.164644 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.164648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.165715 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.165719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.167570 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.167581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.168743 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.168750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.168894 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.168928 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.169046 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.169122 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260929 10:23:51.169277 29335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91648e54c8834adeb58409210b1d6d9b" candidate_uuid: "448c674331194ea8afa98a5290d97334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" is_pre_election: true
I20260929 10:23:51.169422 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.164025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:56445 (local address 127.21.33.130:36499)
0929 10:23:51.164188 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.164193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.164268 (+ 75us) server_negotiation.cc:407] Connection header received
0929 10:23:51.164368 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.164370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.164437 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.164513 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.164801 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.165587 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.167719 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.168458 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.168593 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.168659 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.169105 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.169253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 10:23:51.169654 29335 raft_consensus.cc:2454] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448c674331194ea8afa98a5290d97334 in term 0.
I20260929 10:23:51.170338 29547 leader_election.cc:302] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448c674331194ea8afa98a5290d97334, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.170562 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91648e54c8834adeb58409210b1d6d9b" candidate_uuid: "448c674331194ea8afa98a5290d97334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54" is_pre_election: true
I20260929 10:23:51.170837 29475 raft_consensus.cc:2454] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448c674331194ea8afa98a5290d97334 in term 0.
I20260929 10:23:51.170859 29695 raft_consensus.cc:2781] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:51.170953 29695 raft_consensus.cc:482] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:51.170997 29695 raft_consensus.cc:3037] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.172362 29695 raft_consensus.cc:504] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.172762 29695 leader_election.cc:288] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [CANDIDATE]: Term 1 election: Requested vote from peers b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021), 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499)
I20260929 10:23:51.173117 29335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91648e54c8834adeb58409210b1d6d9b" candidate_uuid: "448c674331194ea8afa98a5290d97334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b327f9f45d42f19737f9ae6e3ca807"
I20260929 10:23:51.173149 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91648e54c8834adeb58409210b1d6d9b" candidate_uuid: "448c674331194ea8afa98a5290d97334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54"
I20260929 10:23:51.173273 29335 raft_consensus.cc:3037] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.173290 29475 raft_consensus.cc:3037] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.174640 29335 raft_consensus.cc:2454] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448c674331194ea8afa98a5290d97334 in term 1.
I20260929 10:23:51.174980 29475 raft_consensus.cc:2454] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448c674331194ea8afa98a5290d97334 in term 1.
I20260929 10:23:51.175282 29547 leader_election.cc:302] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448c674331194ea8afa98a5290d97334, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.175462 29695 raft_consensus.cc:2781] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:51.175700 29695 raft_consensus.cc:686] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 1 LEADER]: Becoming Leader. State: Replica: 448c674331194ea8afa98a5290d97334, State: Running, Role: LEADER
I20260929 10:23:51.176048 29695 consensus_queue.cc:237] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.179231 29177 catalog_manager.cc:5495] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 reported cstate change: term changed from 0 to 1, leader changed from <none> to 448c674331194ea8afa98a5290d97334 (127.21.33.131). New cstate: current_term: 1 leader_uuid: "448c674331194ea8afa98a5290d97334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } health_report { overall_health: HEALTHY } } }
I20260929 10:23:51.194736 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.191598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.21.33.131:38549)
0929 10:23:51.191703 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.191706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.191719 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:51.191837 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.191839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.191888 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.191969 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:51.192158 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.192634 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.193489 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.193709 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.193797 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.194003 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.194392 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.194473 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.194528 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.194571 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 10:23:51.200421 29335 raft_consensus.cc:1261] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Refusing update from remote peer 448c674331194ea8afa98a5290d97334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.200421 29475 raft_consensus.cc:1261] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Refusing update from remote peer 448c674331194ea8afa98a5290d97334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.201128 29699 consensus_queue.cc:1035] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.201366 29695 consensus_queue.cc:1035] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.208937 29703 mvcc.cc:205] Tried to move back new op lower bound from 7334614758189441024 to 7334614758099861504. Current Snapshot: MvccSnapshot[applied={T|T < 7334614758189441024}]
I20260929 10:23:51.210450 29702 mvcc.cc:205] Tried to move back new op lower bound from 7334614758189441024 to 7334614758099861504. Current Snapshot: MvccSnapshot[applied={T|T < 7334614758189441024 or (T in {7334614758189441024})}]
I20260929 10:23:51.211284 29704 mvcc.cc:205] Tried to move back new op lower bound from 7334614758189441024 to 7334614758099861504. Current Snapshot: MvccSnapshot[applied={T|T < 7334614758189441024}]
I20260929 10:23:51.216665 29177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:51.216826 29177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:51.219208 29177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Sending DeleteTablet for 3 replicas of tablet 91648e54c8834adeb58409210b1d6d9b
I20260929 10:23:51.220017 29595 tablet_service.cc:1514] Processing DeleteTablet for tablet 91648e54c8834adeb58409210b1d6d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:60866
I20260929 10:23:51.220021 29455 tablet_service.cc:1514] Processing DeleteTablet for tablet 91648e54c8834adeb58409210b1d6d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:34518
I20260929 10:23:51.220067 29315 tablet_service.cc:1514] Processing DeleteTablet for tablet 91648e54c8834adeb58409210b1d6d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:53708
I20260929 10:23:51.220504 29712 tablet_replica.cc:330] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: stopping tablet replica
I20260929 10:23:51.220805 29712 raft_consensus.cc:2229] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:51.220985 29713 tablet_replica.cc:330] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: stopping tablet replica
I20260929 10:23:51.221125 29712 raft_consensus.cc:2258] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.221261 29177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:23:51.221495 29711 tablet_replica.cc:330] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: stopping tablet replica
W20260929 10:23:51.221663 29177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:51.221751 29711 raft_consensus.cc:2229] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:51.222105 29711 raft_consensus.cc:2258] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.222497 29713 raft_consensus.cc:2229] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:51.222785 29713 raft_consensus.cc:2258] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.223624 29712 ts_tablet_manager.cc:1905] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.226529 29455 tablet_service.cc:1467] Processing CreateTablet for tablet 358adcee0bcd46e99061971173db30a8 (DEFAULT_TABLE table=test-table [id=cd575df76acd415db02a77ce198a587b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.226589 29315 tablet_service.cc:1467] Processing CreateTablet for tablet 358adcee0bcd46e99061971173db30a8 (DEFAULT_TABLE table=test-table [id=cd575df76acd415db02a77ce198a587b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.226807 29455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358adcee0bcd46e99061971173db30a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.226855 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358adcee0bcd46e99061971173db30a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.227662 29712 ts_tablet_manager.cc:1918] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:51.227743 29712 log.cc:1196] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/91648e54c8834adeb58409210b1d6d9b
I20260929 10:23:51.228075 29712 ts_tablet_manager.cc:1939] T 91648e54c8834adeb58409210b1d6d9b P 992dcee63bc743bb98db0cdb48aace54: Deleting consensus metadata
I20260929 10:23:51.229103 29690 tablet_bootstrap.cc:492] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Bootstrap starting.
I20260929 10:23:51.229279 29691 tablet_bootstrap.cc:492] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Bootstrap starting.
I20260929 10:23:51.229331 29163 catalog_manager.cc:4841] TS 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499): tablet 91648e54c8834adeb58409210b1d6d9b (table test-table [id=83634a7d9f9d4b76a782bc44d59ea6db]) successfully deleted
I20260929 10:23:51.229561 29595 tablet_service.cc:1467] Processing CreateTablet for tablet 358adcee0bcd46e99061971173db30a8 (DEFAULT_TABLE table=test-table [id=cd575df76acd415db02a77ce198a587b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.229816 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358adcee0bcd46e99061971173db30a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.230013 29713 ts_tablet_manager.cc:1905] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.230170 29690 tablet_bootstrap.cc:654] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.230189 29691 tablet_bootstrap.cc:654] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.232276 29689 tablet_bootstrap.cc:492] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Bootstrap starting.
I20260929 10:23:51.232420 29713 ts_tablet_manager.cc:1918] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:51.232513 29713 log.cc:1196] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/91648e54c8834adeb58409210b1d6d9b
I20260929 10:23:51.232980 29691 tablet_bootstrap.cc:492] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: No bootstrap required, opened a new log
I20260929 10:23:51.233060 29691 ts_tablet_manager.cc:1397] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 10:23:51.233217 29713 ts_tablet_manager.cc:1939] T 91648e54c8834adeb58409210b1d6d9b P b2b327f9f45d42f19737f9ae6e3ca807: Deleting consensus metadata
I20260929 10:23:51.233407 29689 tablet_bootstrap.cc:654] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.233516 29691 raft_consensus.cc:348] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.233623 29691 raft_consensus.cc:374] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.233673 29691 raft_consensus.cc:729] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992dcee63bc743bb98db0cdb48aace54, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.233785 29691 consensus_queue.cc:260] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.234038 29691 ts_tablet_manager.cc:1428] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:51.234356 29711 ts_tablet_manager.cc:1905] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.234544 29163 catalog_manager.cc:4841] TS b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021): tablet 91648e54c8834adeb58409210b1d6d9b (table test-table [id=83634a7d9f9d4b76a782bc44d59ea6db]) successfully deleted
I20260929 10:23:51.236800 29711 ts_tablet_manager.cc:1918] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:51.236915 29711 log.cc:1196] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/91648e54c8834adeb58409210b1d6d9b
I20260929 10:23:51.237190 29711 ts_tablet_manager.cc:1939] T 91648e54c8834adeb58409210b1d6d9b P 448c674331194ea8afa98a5290d97334: Deleting consensus metadata
I20260929 10:23:51.237995 29689 tablet_bootstrap.cc:492] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: No bootstrap required, opened a new log
I20260929 10:23:51.238080 29689 ts_tablet_manager.cc:1397] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260929 10:23:51.238343 29163 catalog_manager.cc:4841] TS 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549): tablet 91648e54c8834adeb58409210b1d6d9b (table test-table [id=83634a7d9f9d4b76a782bc44d59ea6db]) successfully deleted
I20260929 10:23:51.238521 29689 raft_consensus.cc:348] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.238637 29689 raft_consensus.cc:374] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.238682 29689 raft_consensus.cc:729] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448c674331194ea8afa98a5290d97334, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.238819 29689 consensus_queue.cc:260] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.239089 29689 ts_tablet_manager.cc:1428] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:51.239867 29690 tablet_bootstrap.cc:492] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: No bootstrap required, opened a new log
I20260929 10:23:51.239955 29690 ts_tablet_manager.cc:1397] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260929 10:23:51.240427 29690 raft_consensus.cc:348] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.240528 29690 raft_consensus.cc:374] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.240550 29690 raft_consensus.cc:729] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.240671 29690 consensus_queue.cc:260] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.240924 29690 ts_tablet_manager.cc:1428] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260929 10:23:51.293040 29662 tablet.cc:2369] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:23:51.323190 29522 tablet.cc:2369] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:23:51.361677 29382 tablet.cc:2369] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:23:51.443151 29696 raft_consensus.cc:482] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:51.443346 29696 raft_consensus.cc:504] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.444501 29696 leader_election.cc:288] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:51.448642 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.444857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:60895 (local address 127.21.33.129:35021)
0929 10:23:51.444996 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.445000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.445147 (+ 147us) server_negotiation.cc:407] Connection header received
0929 10:23:51.445343 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.445347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.445423 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.445543 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.445916 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.446545 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.447679 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.448080 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.448214 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.448279 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.448443 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.448515 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:23:51.448652 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.444769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35021 (local address 127.21.33.130:60895)
0929 10:23:51.445028 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.445044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.445158 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.445656 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.445658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.445676 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.445788 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.445793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.446662 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.446668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.447551 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.447561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.448205 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.448209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.448309 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.448340 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.448429 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.448510 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260929 10:23:51.448869 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.445308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:35371 (local address 127.21.33.131:38549)
0929 10:23:51.445444 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.445448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.445624 (+ 176us) server_negotiation.cc:407] Connection header received
0929 10:23:51.445726 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.445731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.445782 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.445875 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.446143 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.446752 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.447941 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.448335 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.448460 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.448515 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.448691 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.448761 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:23:51.448879 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.445159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:38549 (local address 127.21.33.130:35371)
0929 10:23:51.445534 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.445548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.445640 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.445884 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.445888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.445902 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.446031 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.446044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.446860 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.446866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.447826 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.447836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.448449 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.448453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.448560 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.448581 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.448675 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.448741 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":90,"threads_started":1}
I20260929 10:23:51.449374 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358adcee0bcd46e99061971173db30a8" candidate_uuid: "992dcee63bc743bb98db0cdb48aace54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334" is_pre_election: true
I20260929 10:23:51.449596 29335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358adcee0bcd46e99061971173db30a8" candidate_uuid: "992dcee63bc743bb98db0cdb48aace54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" is_pre_election: true
I20260929 10:23:51.449622 29615 raft_consensus.cc:2454] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 992dcee63bc743bb98db0cdb48aace54 in term 0.
I20260929 10:23:51.449728 29335 raft_consensus.cc:2454] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 992dcee63bc743bb98db0cdb48aace54 in term 0.
I20260929 10:23:51.450052 29407 leader_election.cc:302] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 992dcee63bc743bb98db0cdb48aace54, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.450314 29696 raft_consensus.cc:2781] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:51.450367 29696 raft_consensus.cc:482] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:51.450394 29696 raft_consensus.cc:3037] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.451344 29696 raft_consensus.cc:504] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.451676 29696 leader_election.cc:288] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [CANDIDATE]: Term 1 election: Requested vote from peers b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:51.452033 29335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358adcee0bcd46e99061971173db30a8" candidate_uuid: "992dcee63bc743bb98db0cdb48aace54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b327f9f45d42f19737f9ae6e3ca807"
I20260929 10:23:51.452069 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358adcee0bcd46e99061971173db30a8" candidate_uuid: "992dcee63bc743bb98db0cdb48aace54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334"
I20260929 10:23:51.452159 29335 raft_consensus.cc:3037] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.452160 29615 raft_consensus.cc:3037] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.453244 29615 raft_consensus.cc:2454] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992dcee63bc743bb98db0cdb48aace54 in term 1.
I20260929 10:23:51.453244 29335 raft_consensus.cc:2454] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992dcee63bc743bb98db0cdb48aace54 in term 1.
I20260929 10:23:51.453519 29407 leader_election.cc:302] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 992dcee63bc743bb98db0cdb48aace54, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.453729 29696 raft_consensus.cc:2781] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:51.453950 29696 raft_consensus.cc:686] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 1 LEADER]: Becoming Leader. State: Replica: 992dcee63bc743bb98db0cdb48aace54, State: Running, Role: LEADER
I20260929 10:23:51.454201 29696 consensus_queue.cc:237] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.456724 29178 catalog_manager.cc:5495] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130). New cstate: current_term: 1 leader_uuid: "992dcee63bc743bb98db0cdb48aace54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } health_report { overall_health: UNKNOWN } } }
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677430, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.190@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:51.502610 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.494949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.21.33.190:34155)
0929 10:23:51.495067 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.495071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.495258 (+ 187us) server_negotiation.cc:407] Connection header received
0929 10:23:51.495355 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.495357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.495394 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.495470 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:51.496106 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.496608 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.497422 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.497620 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.497695 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.499903 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:51.499907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:51.499909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:51.499928 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:51.501134 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.501589 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.501591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.501593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.501663 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.501891 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.501893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.501894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.502046 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:51.502119 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.502377 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.502490 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":43}
I20260929 10:23:51.502619 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.494887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34155 (local address 127.0.0.1:34892)
0929 10:23:51.495168 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.495185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.495290 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.495506 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.495508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.495904 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:51.496014 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.496020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.496695 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.496697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.497326 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.497335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.497711 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.497714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.497812 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.498420 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:51.498432 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:51.499775 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:51.501276 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:51.501285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:51.501290 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:51.501506 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:51.501741 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:51.501743 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:51.501745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:51.501820 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:51.502131 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:51.502136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:51.502230 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.502326 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.502438 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
I20260929 10:23:51.506464 29178 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:34892:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 10:23:51.507201 29178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:51.511848 29315 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4a8594c6324142b2356c9931d3a4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8986cb4213d4e4c93e487e55efb6b12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:51.511848 29455 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4a8594c6324142b2356c9931d3a4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8986cb4213d4e4c93e487e55efb6b12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:51.511914 29595 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4a8594c6324142b2356c9931d3a4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e8986cb4213d4e4c93e487e55efb6b12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 10:23:51.512256 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4a8594c6324142b2356c9931d3a4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.512256 29455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4a8594c6324142b2356c9931d3a4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.512281 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4a8594c6324142b2356c9931d3a4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.514752 29689 tablet_bootstrap.cc:492] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: Bootstrap starting.
I20260929 10:23:51.515028 29690 tablet_bootstrap.cc:492] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: Bootstrap starting.
I20260929 10:23:51.515194 29691 tablet_bootstrap.cc:492] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: Bootstrap starting.
I20260929 10:23:51.515705 29689 tablet_bootstrap.cc:654] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.516023 29690 tablet_bootstrap.cc:654] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.516116 29691 tablet_bootstrap.cc:654] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.516920 29689 tablet_bootstrap.cc:492] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: No bootstrap required, opened a new log
I20260929 10:23:51.517023 29689 ts_tablet_manager.cc:1397] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 10:23:51.517289 29690 tablet_bootstrap.cc:492] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: No bootstrap required, opened a new log
I20260929 10:23:51.517383 29690 ts_tablet_manager.cc:1397] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:51.517495 29689 raft_consensus.cc:348] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.517637 29689 raft_consensus.cc:374] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.517675 29689 raft_consensus.cc:729] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448c674331194ea8afa98a5290d97334, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.517807 29689 consensus_queue.cc:260] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.517845 29690 raft_consensus.cc:348] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.517963 29690 raft_consensus.cc:374] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.518008 29690 raft_consensus.cc:729] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.518076 29689 ts_tablet_manager.cc:1428] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:51.518134 29690 consensus_queue.cc:260] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.518258 29699 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.518414 29699 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:51.518412 29690 ts_tablet_manager.cc:1428] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:51.518596 29697 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.518743 29697 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:51.519574 29691 tablet_bootstrap.cc:492] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: No bootstrap required, opened a new log
I20260929 10:23:51.519640 29691 ts_tablet_manager.cc:1397] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 10:23:51.519984 29691 raft_consensus.cc:348] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.520087 29691 raft_consensus.cc:374] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.520120 29691 raft_consensus.cc:729] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992dcee63bc743bb98db0cdb48aace54, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.520262 29691 consensus_queue.cc:260] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.520574 29691 ts_tablet_manager.cc:1428] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:51.520807 29696 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.520925 29696 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:51.550591 29697 raft_consensus.cc:482] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:51.550729 29697 raft_consensus.cc:504] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.551759 29697 leader_election.cc:288] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:51.557279 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.552114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:52769 (local address 127.21.33.130:36499)
0929 10:23:51.552254 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.552259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.552443 (+ 184us) server_negotiation.cc:407] Connection header received
0929 10:23:51.552685 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.552690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.552741 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.552792 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.553263 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.554064 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.555994 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.556569 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.556691 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.556745 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.557058 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.557127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:23:51.557279 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.552519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:39115 (local address 127.21.33.131:38549)
0929 10:23:51.552637 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.552643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.552873 (+ 230us) server_negotiation.cc:407] Connection header received
0929 10:23:51.552989 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.552991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.553030 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.553094 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:23:51.553391 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.554109 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.555956 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.556580 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.556692 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.556753 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.557058 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.557127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:23:51.557335 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.552011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:36499 (local address 127.21.33.129:52769)
0929 10:23:51.552352 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.552369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.552530 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.552943 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.552946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.552965 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.553105 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.553114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.554198 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.554201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.555829 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.555840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.556693 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.556697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.556887 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.556955 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.557030 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.557132 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":284,"thread_start_us":117,"threads_started":1}
I20260929 10:23:51.557335 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.552428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:38549 (local address 127.21.33.129:39115)
0929 10:23:51.552803 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.552816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.552901 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.553129 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.553132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.553152 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:23:51.553293 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.553298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.554199 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.554202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.555830 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.555840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.556693 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.556697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.556877 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.556950 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.557030 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.557128 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":211,"thread_start_us":90,"threads_started":1}
I20260929 10:23:51.557868 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a8594c6324142b2356c9931d3a4a8" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334" is_pre_election: true
I20260929 10:23:51.557873 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a8594c6324142b2356c9931d3a4a8" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54" is_pre_election: true
I20260929 10:23:51.557998 29615 raft_consensus.cc:2454] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 0.
I20260929 10:23:51.558030 29475 raft_consensus.cc:2454] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 0.
I20260929 10:23:51.558382 29267 leader_election.cc:302] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448c674331194ea8afa98a5290d97334, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.558660 29697 raft_consensus.cc:2781] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:51.558714 29697 raft_consensus.cc:482] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:51.558741 29697 raft_consensus.cc:3037] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.559751 29697 raft_consensus.cc:504] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.560092 29697 leader_election.cc:288] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 election: Requested vote from peers 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:51.560437 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a8594c6324142b2356c9931d3a4a8" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54"
I20260929 10:23:51.560494 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a8594c6324142b2356c9931d3a4a8" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334"
I20260929 10:23:51.560601 29475 raft_consensus.cc:3037] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.560612 29615 raft_consensus.cc:3037] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.561555 29615 raft_consensus.cc:2454] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 1.
I20260929 10:23:51.561604 29475 raft_consensus.cc:2454] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 1.
I20260929 10:23:51.561817 29267 leader_election.cc:302] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448c674331194ea8afa98a5290d97334, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.561978 29697 raft_consensus.cc:2781] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:51.562184 29697 raft_consensus.cc:686] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 LEADER]: Becoming Leader. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Running, Role: LEADER
I20260929 10:23:51.562500 29697 consensus_queue.cc:237] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.563872 29732 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: TxnStatusTablet state changed. Reason: New leader b2b327f9f45d42f19737f9ae6e3ca807. Latest consensus state: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.563979 29732 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:51.564252 29734 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 10:23:51.564399 29734 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 10:23:51.565011 29177 catalog_manager.cc:5495] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129). New cstate: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:51.585079 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.580882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35021 (local address 127.0.0.1:53714)
0929 10:23:51.581143 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.581155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.581319 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.581639 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.581641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.581648 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:51.581735 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.581742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.582710 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.582712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.583442 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.583452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.584030 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.584033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.584093 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.584134 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.584845 (+ 711us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.584856 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.584902 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.584969 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 10:23:51.585103 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.581009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.21.33.129:35021)
0929 10:23:51.581138 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.581142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.581214 (+ 72us) server_negotiation.cc:407] Connection header received
0929 10:23:51.581452 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.581456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.581518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.581619 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:51.581834 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.582611 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.583551 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.583922 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.584022 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.584261 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.584741 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.584860 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.584925 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.584977 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:23:51.587756 29475 raft_consensus.cc:1261] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Refusing update from remote peer b2b327f9f45d42f19737f9ae6e3ca807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.587981 29615 raft_consensus.cc:1261] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Refusing update from remote peer b2b327f9f45d42f19737f9ae6e3ca807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.588218 29697 consensus_queue.cc:1035] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.588450 29732 consensus_queue.cc:1035] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.589804 29696 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: TxnStatusTablet state changed. Reason: New leader b2b327f9f45d42f19737f9ae6e3ca807. Latest consensus state: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.589928 29696 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:51.589943 29699 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: TxnStatusTablet state changed. Reason: New leader b2b327f9f45d42f19737f9ae6e3ca807. Latest consensus state: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.590067 29699 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:51.591893 29732 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.592018 29732 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:51.593864 29697 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.593986 29697 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P b2b327f9f45d42f19737f9ae6e3ca807: This TxnStatusTablet replica's current role is: LEADER
I20260929 10:23:51.594436 29696 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.594551 29696 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P 992dcee63bc743bb98db0cdb48aace54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:51.596701 29699 tablet_replica.cc:439] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } } }
I20260929 10:23:51.596817 29699 tablet_replica.cc:442] T 2c4a8594c6324142b2356c9931d3a4a8 P 448c674331194ea8afa98a5290d97334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 10:23:51.601470 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.597892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.21.33.130:36499)
0929 10:23:51.598001 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.598006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.598020 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:51.598106 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.598109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.598169 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.598292 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:51.598542 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.599202 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.600018 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.600257 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.600364 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.600633 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.601053 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.601181 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.601286 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.601340 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:23:51.612663 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.608602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35021 (local address 127.21.33.130:41837)
0929 10:23:51.608899 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.608914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.609011 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.609296 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.609299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.609306 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:51.609423 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.609428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.610231 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.610243 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.610867 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.610876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.611418 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.611422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.611510 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.611550 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.611891 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.611901 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.611972 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.612051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":121,"threads_started":1}
I20260929 10:23:51.612226 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.608921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:41837 (local address 127.21.33.129:35021)
0929 10:23:51.609042 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.609048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.609074 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:23:51.609125 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.609128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.609167 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.609285 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:51.609528 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.610121 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.610989 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.611302 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.611414 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.611684 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.611770 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.611853 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.611995 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.612043 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:23:51.618988 29614 raft_consensus.cc:1261] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Refusing update from remote peer 992dcee63bc743bb98db0cdb48aace54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.619474 29719 consensus_queue.cc:1035] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.620298 29335 raft_consensus.cc:1261] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Refusing update from remote peer 992dcee63bc743bb98db0cdb48aace54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.620932 29719 consensus_queue.cc:1035] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.644455 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.640560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:33893 (local address 127.21.33.130:36499)
0929 10:23:51.640678 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.640683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.640907 (+ 224us) server_negotiation.cc:407] Connection header received
0929 10:23:51.641021 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.641023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.641061 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.641120 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:51.641413 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.642104 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.643405 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.643635 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.643745 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.644027 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.644102 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.644179 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.644271 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.644320 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:23:51.644474 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.640456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:36499 (local address 127.21.33.129:33893)
0929 10:23:51.640817 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:51.640832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:51.640936 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:51.641151 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:51.641154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:51.641162 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:51.641308 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.641315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.642344 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.642349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.643287 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:51.643297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.643752 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.643756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:51.643854 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.643895 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.644194 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.644201 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 10:23:51.644260 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 10:23:51.644335 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":126,"threads_started":1}
I20260929 10:23:51.667567 29178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:51.667732 29178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:51.669950 29178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Sending DeleteTablet for 3 replicas of tablet 358adcee0bcd46e99061971173db30a8
I20260929 10:23:51.670683 29455 tablet_service.cc:1514] Processing DeleteTablet for tablet 358adcee0bcd46e99061971173db30a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:34518
I20260929 10:23:51.670877 29712 tablet_replica.cc:330] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: stopping tablet replica
I20260929 10:23:51.670881 29595 tablet_service.cc:1514] Processing DeleteTablet for tablet 358adcee0bcd46e99061971173db30a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:60866
I20260929 10:23:51.671006 29712 raft_consensus.cc:2229] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:51.671032 29711 tablet_replica.cc:330] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: stopping tablet replica
I20260929 10:23:51.671131 29711 raft_consensus.cc:2229] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:51.671226 29711 raft_consensus.cc:2258] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.671310 29712 raft_consensus.cc:2258] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.671679 29315 tablet_service.cc:1514] Processing DeleteTablet for tablet 358adcee0bcd46e99061971173db30a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:53708
I20260929 10:23:51.672245 29711 ts_tablet_manager.cc:1905] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.675153 29713 tablet_replica.cc:330] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: stopping tablet replica
I20260929 10:23:51.675312 29713 raft_consensus.cc:2229] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:51.675446 29713 raft_consensus.cc:2258] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.675515 29711 ts_tablet_manager.cc:1918] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:51.676218 29712 ts_tablet_manager.cc:1905] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.676337 29713 ts_tablet_manager.cc:1905] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.679181 29713 ts_tablet_manager.cc:1918] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:51.679311 29713 log.cc:1196] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/358adcee0bcd46e99061971173db30a8
I20260929 10:23:51.679311 29712 ts_tablet_manager.cc:1918] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:51.679395 29712 log.cc:1196] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/358adcee0bcd46e99061971173db30a8
I20260929 10:23:51.679692 29712 ts_tablet_manager.cc:1939] T 358adcee0bcd46e99061971173db30a8 P 992dcee63bc743bb98db0cdb48aace54: Deleting consensus metadata
I20260929 10:23:51.675755 29711 log.cc:1196] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/358adcee0bcd46e99061971173db30a8
I20260929 10:23:51.679943 29713 ts_tablet_manager.cc:1939] T 358adcee0bcd46e99061971173db30a8 P b2b327f9f45d42f19737f9ae6e3ca807: Deleting consensus metadata
I20260929 10:23:51.680241 29711 ts_tablet_manager.cc:1939] T 358adcee0bcd46e99061971173db30a8 P 448c674331194ea8afa98a5290d97334: Deleting consensus metadata
I20260929 10:23:51.680446 29163 catalog_manager.cc:4841] TS 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499): tablet 358adcee0bcd46e99061971173db30a8 (table test-table [id=cd575df76acd415db02a77ce198a587b]) successfully deleted
I20260929 10:23:51.681345 29163 catalog_manager.cc:4841] TS 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549): tablet 358adcee0bcd46e99061971173db30a8 (table test-table [id=cd575df76acd415db02a77ce198a587b]) successfully deleted
I20260929 10:23:51.681523 29163 catalog_manager.cc:4841] TS b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021): tablet 358adcee0bcd46e99061971173db30a8 (table test-table [id=cd575df76acd415db02a77ce198a587b]) successfully deleted
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:23:51.714684 29228 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:51.706467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.21.33.190:34155)
0929 10:23:51.706608 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.706612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.706951 (+ 339us) server_negotiation.cc:407] Connection header received
0929 10:23:51.707170 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.707174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.707230 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.707324 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:51.708224 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.709056 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.710354 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.710701 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.710813 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.714548 (+ 3735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34902: BlockingRecv error: recv got EOF from 127.0.0.1:34902 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 10:23:51 dist-test-slave-2rbl krb5kdc[29133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM
I20260929 10:23:51.727025 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.716676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.21.33.190:34155)
0929 10:23:51.716801 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.716805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.716922 (+ 117us) server_negotiation.cc:407] Connection header received
0929 10:23:51.717033 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.717036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.717083 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.717165 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:51.717937 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.718725 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.719868 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.720213 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.720318 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.723257 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:51.723262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:51.723265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:51.723294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:51.725064 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.725584 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.725588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.725590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.725684 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:51.726022 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:51.726026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:51.726027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:51.726260 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:51.726397 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.726673 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.726805 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:23:51.730206 29178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:23:51.730715 29178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:51.736172 29315 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf634ad4a35464d9af9627107a84232 (DEFAULT_TABLE table=test-table [id=aa493dc682cd4dd294c811b7e4b0bf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.736496 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf634ad4a35464d9af9627107a84232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.739025 29690 tablet_bootstrap.cc:492] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: Bootstrap starting.
I20260929 10:23:51.739210 29455 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf634ad4a35464d9af9627107a84232 (DEFAULT_TABLE table=test-table [id=aa493dc682cd4dd294c811b7e4b0bf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.739472 29455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf634ad4a35464d9af9627107a84232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.739938 29690 tablet_bootstrap.cc:654] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.741048 29690 tablet_bootstrap.cc:492] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: No bootstrap required, opened a new log
I20260929 10:23:51.741122 29690 ts_tablet_manager.cc:1397] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:51.741539 29690 raft_consensus.cc:348] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.741645 29690 raft_consensus.cc:374] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.741675 29690 raft_consensus.cc:729] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.741786 29690 consensus_queue.cc:260] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.742024 29690 ts_tablet_manager.cc:1428] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:51.742074 29595 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf634ad4a35464d9af9627107a84232 (DEFAULT_TABLE table=test-table [id=aa493dc682cd4dd294c811b7e4b0bf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.742336 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf634ad4a35464d9af9627107a84232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.742830 29691 tablet_bootstrap.cc:492] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: Bootstrap starting.
I20260929 10:23:51.743746 29691 tablet_bootstrap.cc:654] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.744704 29689 tablet_bootstrap.cc:492] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: Bootstrap starting.
I20260929 10:23:51.745549 29689 tablet_bootstrap.cc:654] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.746624 29689 tablet_bootstrap.cc:492] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: No bootstrap required, opened a new log
I20260929 10:23:51.746728 29689 ts_tablet_manager.cc:1397] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:51.747167 29689 raft_consensus.cc:348] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.747303 29689 raft_consensus.cc:374] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.747350 29689 raft_consensus.cc:729] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448c674331194ea8afa98a5290d97334, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.747485 29689 consensus_queue.cc:260] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.747776 29689 ts_tablet_manager.cc:1428] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:23:51.747969 29691 tablet_bootstrap.cc:492] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: No bootstrap required, opened a new log
I20260929 10:23:51.748050 29691 ts_tablet_manager.cc:1397] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 10:23:51.748549 29691 raft_consensus.cc:348] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.748680 29691 raft_consensus.cc:374] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.748723 29691 raft_consensus.cc:729] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992dcee63bc743bb98db0cdb48aace54, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.748881 29691 consensus_queue.cc:260] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.749130 29691 ts_tablet_manager.cc:1428] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:23:51.886261 29739 raft_consensus.cc:482] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:51.886404 29739 raft_consensus.cc:504] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.886760 29739 leader_election.cc:288] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:51.887306 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf634ad4a35464d9af9627107a84232" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54" is_pre_election: true
I20260929 10:23:51.887306 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf634ad4a35464d9af9627107a84232" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334" is_pre_election: true
I20260929 10:23:51.887465 29615 raft_consensus.cc:2454] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 0.
I20260929 10:23:51.887465 29475 raft_consensus.cc:2454] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 0.
I20260929 10:23:51.887768 29267 leader_election.cc:302] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 992dcee63bc743bb98db0cdb48aace54, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.887943 29739 raft_consensus.cc:2781] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:51.888005 29739 raft_consensus.cc:482] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:51.888047 29739 raft_consensus.cc:3037] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.889050 29739 raft_consensus.cc:504] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.889441 29739 leader_election.cc:288] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 election: Requested vote from peers 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:51.889818 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf634ad4a35464d9af9627107a84232" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334"
I20260929 10:23:51.889818 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf634ad4a35464d9af9627107a84232" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54"
I20260929 10:23:51.889946 29475 raft_consensus.cc:3037] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.889971 29615 raft_consensus.cc:3037] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:51.891079 29615 raft_consensus.cc:2454] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 1.
I20260929 10:23:51.891079 29475 raft_consensus.cc:2454] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 1.
I20260929 10:23:51.891381 29267 leader_election.cc:302] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448c674331194ea8afa98a5290d97334, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:51.891559 29739 raft_consensus.cc:2781] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:51.891649 29739 raft_consensus.cc:686] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 LEADER]: Becoming Leader. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Running, Role: LEADER
I20260929 10:23:51.891816 29739 consensus_queue.cc:237] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.893512 29178 catalog_manager.cc:5495] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129). New cstate: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:51.904162 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:51.901566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.21.33.129:35021)
0929 10:23:51.901678 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:51.901681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:51.901693 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:51.901800 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:51.901802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:51.901843 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:51.901905 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:51.902105 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.902583 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.903362 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:51.903562 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:51.903634 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:51.903836 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:51.903901 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:51.903953 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:51.904027 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 10:23:51.904063 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:23:51.906309 29615 raft_consensus.cc:1261] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Refusing update from remote peer b2b327f9f45d42f19737f9ae6e3ca807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.906309 29475 raft_consensus.cc:1261] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Refusing update from remote peer b2b327f9f45d42f19737f9ae6e3ca807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:51.906749 29732 consensus_queue.cc:1035] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.906890 29739 consensus_queue.cc:1035] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:51.916515 29178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:51.916643 29178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:51.918641 29178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Sending DeleteTablet for 3 replicas of tablet 7bf634ad4a35464d9af9627107a84232
I20260929 10:23:51.919164 29315 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf634ad4a35464d9af9627107a84232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:53708
I20260929 10:23:51.919303 29595 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf634ad4a35464d9af9627107a84232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:60866
I20260929 10:23:51.919374 29713 tablet_replica.cc:330] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: stopping tablet replica
I20260929 10:23:51.919492 29711 tablet_replica.cc:330] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: stopping tablet replica
I20260929 10:23:51.919493 29713 raft_consensus.cc:2229] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:51.919616 29711 raft_consensus.cc:2229] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:51.919746 29711 raft_consensus.cc:2258] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.919783 29713 raft_consensus.cc:2258] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.919811 29455 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf634ad4a35464d9af9627107a84232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:51 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:34518
I20260929 10:23:51.920017 29712 tablet_replica.cc:330] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: stopping tablet replica
I20260929 10:23:51.920130 29712 raft_consensus.cc:2229] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:51.920279 29712 raft_consensus.cc:2258] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:51.920996 29711 ts_tablet_manager.cc:1905] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.921082 29712 ts_tablet_manager.cc:1905] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.921123 29713 ts_tablet_manager.cc:1905] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:51.923396 29711 ts_tablet_manager.cc:1918] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:51.923396 29712 ts_tablet_manager.cc:1918] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:51.923465 29711 log.cc:1196] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/7bf634ad4a35464d9af9627107a84232
I20260929 10:23:51.923465 29712 log.cc:1196] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/7bf634ad4a35464d9af9627107a84232
I20260929 10:23:51.923501 29713 ts_tablet_manager.cc:1918] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:23:51.923568 29713 log.cc:1196] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/7bf634ad4a35464d9af9627107a84232
I20260929 10:23:51.923727 29712 ts_tablet_manager.cc:1939] T 7bf634ad4a35464d9af9627107a84232 P 992dcee63bc743bb98db0cdb48aace54: Deleting consensus metadata
I20260929 10:23:51.923727 29711 ts_tablet_manager.cc:1939] T 7bf634ad4a35464d9af9627107a84232 P 448c674331194ea8afa98a5290d97334: Deleting consensus metadata
I20260929 10:23:51.923807 29713 ts_tablet_manager.cc:1939] T 7bf634ad4a35464d9af9627107a84232 P b2b327f9f45d42f19737f9ae6e3ca807: Deleting consensus metadata
I20260929 10:23:51.924103 29178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 10:23:51.924503 29163 catalog_manager.cc:4841] TS 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549): tablet 7bf634ad4a35464d9af9627107a84232 (table test-table [id=aa493dc682cd4dd294c811b7e4b0bf34]) successfully deleted
W20260929 10:23:51.924546 29178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:23:51.924768 29163 catalog_manager.cc:4841] TS b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021): tablet 7bf634ad4a35464d9af9627107a84232 (table test-table [id=aa493dc682cd4dd294c811b7e4b0bf34]) successfully deleted
I20260929 10:23:51.925012 29163 catalog_manager.cc:4841] TS 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499): tablet 7bf634ad4a35464d9af9627107a84232 (table test-table [id=aa493dc682cd4dd294c811b7e4b0bf34]) successfully deleted
I20260929 10:23:51.928923 29315 tablet_service.cc:1467] Processing CreateTablet for tablet ec8426339bbd44439852c808da8dd26d (DEFAULT_TABLE table=test-table [id=ede49ffa55f04b438351ae939939ee51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.928922 29455 tablet_service.cc:1467] Processing CreateTablet for tablet ec8426339bbd44439852c808da8dd26d (DEFAULT_TABLE table=test-table [id=ede49ffa55f04b438351ae939939ee51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.929160 29455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8426339bbd44439852c808da8dd26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.929186 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8426339bbd44439852c808da8dd26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.931238 29691 tablet_bootstrap.cc:492] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: Bootstrap starting.
I20260929 10:23:51.931761 29595 tablet_service.cc:1467] Processing CreateTablet for tablet ec8426339bbd44439852c808da8dd26d (DEFAULT_TABLE table=test-table [id=ede49ffa55f04b438351ae939939ee51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:23:51.932010 29595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8426339bbd44439852c808da8dd26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:51.932032 29691 tablet_bootstrap.cc:654] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.932533 29690 tablet_bootstrap.cc:492] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: Bootstrap starting.
I20260929 10:23:51.933300 29690 tablet_bootstrap.cc:654] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.933614 29691 tablet_bootstrap.cc:492] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: No bootstrap required, opened a new log
I20260929 10:23:51.933681 29691 ts_tablet_manager.cc:1397] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 10:23:51.934154 29689 tablet_bootstrap.cc:492] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: Bootstrap starting.
I20260929 10:23:51.934113 29691 raft_consensus.cc:348] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.934216 29691 raft_consensus.cc:374] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.934237 29690 tablet_bootstrap.cc:492] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: No bootstrap required, opened a new log
I20260929 10:23:51.934247 29691 raft_consensus.cc:729] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992dcee63bc743bb98db0cdb48aace54, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.934304 29690 ts_tablet_manager.cc:1397] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:51.934361 29691 consensus_queue.cc:260] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.934604 29691 ts_tablet_manager.cc:1428] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 10:23:51.934639 29690 raft_consensus.cc:348] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.934732 29690 raft_consensus.cc:374] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.934769 29690 raft_consensus.cc:729] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.934870 29690 consensus_queue.cc:260] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.934907 29689 tablet_bootstrap.cc:654] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:51.935096 29690 ts_tablet_manager.cc:1428] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 10:23:51.936404 29689 tablet_bootstrap.cc:492] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: No bootstrap required, opened a new log
I20260929 10:23:51.936498 29689 ts_tablet_manager.cc:1397] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:51.936906 29689 raft_consensus.cc:348] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.937018 29689 raft_consensus.cc:374] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:51.937069 29689 raft_consensus.cc:729] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448c674331194ea8afa98a5290d97334, State: Initialized, Role: FOLLOWER
I20260929 10:23:51.937182 29689 consensus_queue.cc:260] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:51.937484 29689 ts_tablet_manager.cc:1428] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 10:23:52.030319 29732 raft_consensus.cc:482] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:23:52.030514 29732 raft_consensus.cc:504] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:52.031065 29732 leader_election.cc:288] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:52.031615 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8426339bbd44439852c808da8dd26d" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334" is_pre_election: true
I20260929 10:23:52.031605 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8426339bbd44439852c808da8dd26d" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54" is_pre_election: true
I20260929 10:23:52.031780 29615 raft_consensus.cc:2454] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 0.
I20260929 10:23:52.031777 29475 raft_consensus.cc:2454] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 0.
I20260929 10:23:52.032053 29267 leader_election.cc:302] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448c674331194ea8afa98a5290d97334, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:52.032225 29732 raft_consensus.cc:2781] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:23:52.032292 29732 raft_consensus.cc:482] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:23:52.032336 29732 raft_consensus.cc:3037] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:52.033337 29732 raft_consensus.cc:504] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:52.033690 29732 leader_election.cc:288] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 election: Requested vote from peers 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499), 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549)
I20260929 10:23:52.034039 29475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8426339bbd44439852c808da8dd26d" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "992dcee63bc743bb98db0cdb48aace54"
I20260929 10:23:52.034106 29615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8426339bbd44439852c808da8dd26d" candidate_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448c674331194ea8afa98a5290d97334"
I20260929 10:23:52.034184 29475 raft_consensus.cc:3037] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:52.034230 29615 raft_consensus.cc:3037] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:52.035072 29475 raft_consensus.cc:2454] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 1.
I20260929 10:23:52.035342 29267 leader_election.cc:302] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 992dcee63bc743bb98db0cdb48aace54, b2b327f9f45d42f19737f9ae6e3ca807; no voters:
I20260929 10:23:52.035458 29615 raft_consensus.cc:2454] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b327f9f45d42f19737f9ae6e3ca807 in term 1.
I20260929 10:23:52.035522 29732 raft_consensus.cc:2781] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:52.035614 29732 raft_consensus.cc:686] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 LEADER]: Becoming Leader. State: Replica: b2b327f9f45d42f19737f9ae6e3ca807, State: Running, Role: LEADER
I20260929 10:23:52.035742 29732 consensus_queue.cc:237] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } }
I20260929 10:23:52.037372 29177 catalog_manager.cc:5495] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129). New cstate: current_term: 1 leader_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b327f9f45d42f19737f9ae6e3ca807" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 } health_report { overall_health: UNKNOWN } } }
I20260929 10:23:52.071568 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:52.068918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35021 (local address 127.21.33.129:34829)
0929 10:23:52.069063 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:52.069076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:52.069142 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:52.069391 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:52.069393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:52.069399 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 10:23:52.069477 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:52.069481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:52.070101 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:52.070103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:52.070663 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:52.070670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:52.071048 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:52.071050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:52.071106 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:52.071129 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 10:23:52.071352 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 10:23:52.071356 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:23:52.071398 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 10:23:52.071454 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 10:23:52.071569 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:52.069000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:34829 (local address 127.21.33.129:35021)
0929 10:23:52.069102 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:52.069107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:52.069128 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:23:52.069252 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:52.069256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:52.069306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:52.069367 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:23:52.069544 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:52.070022 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:52.070761 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:52.070959 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:52.071035 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:52.071218 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:23:52.071284 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:23:52.071335 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:52.071414 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 10:23:52.071451 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:23:52.077136 29615 raft_consensus.cc:1261] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Refusing update from remote peer b2b327f9f45d42f19737f9ae6e3ca807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:52.077246 29475 raft_consensus.cc:1261] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Refusing update from remote peer b2b327f9f45d42f19737f9ae6e3ca807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:23:52.077553 29697 consensus_queue.cc:1035] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448c674331194ea8afa98a5290d97334" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:52.077725 29739 consensus_queue.cc:1035] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "992dcee63bc743bb98db0cdb48aace54" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:23:52.106709 29177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:52.106905 29177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:23:52.108789 29177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5e3f06eccd4423cb11d14a158a3e834: Sending DeleteTablet for 3 replicas of tablet ec8426339bbd44439852c808da8dd26d
I20260929 10:23:52.109335 29314 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8426339bbd44439852c808da8dd26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:52 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:53708
I20260929 10:23:52.109402 29455 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8426339bbd44439852c808da8dd26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:52 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:34518
I20260929 10:23:52.109462 29595 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8426339bbd44439852c808da8dd26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:23:52 UTC) from {username='oryx', principal='oryx/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:60866
I20260929 10:23:52.109512 29713 tablet_replica.cc:330] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: stopping tablet replica
I20260929 10:23:52.109565 29712 tablet_replica.cc:330] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: stopping tablet replica
I20260929 10:23:52.109637 29713 raft_consensus.cc:2229] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:23:52.109656 29711 tablet_replica.cc:330] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: stopping tablet replica
I20260929 10:23:52.109683 29712 raft_consensus.cc:2229] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:52.109755 29711 raft_consensus.cc:2229] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:23:52.109776 29712 raft_consensus.cc:2258] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:52.109835 29713 raft_consensus.cc:2258] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:52.109851 29711 raft_consensus.cc:2258] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:23:52.110483 29713 ts_tablet_manager.cc:1905] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:52.110685 29711 ts_tablet_manager.cc:1905] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:52.110821 29712 ts_tablet_manager.cc:1905] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:23:52.112990 29711 ts_tablet_manager.cc:1918] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:52.112990 29713 ts_tablet_manager.cc:1918] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:52.113067 29713 log.cc:1196] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/ec8426339bbd44439852c808da8dd26d
I20260929 10:23:52.113067 29711 log.cc:1196] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/ec8426339bbd44439852c808da8dd26d
I20260929 10:23:52.113039 29712 ts_tablet_manager.cc:1918] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 10:23:52.113122 29712 log.cc:1196] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/ec8426339bbd44439852c808da8dd26d
I20260929 10:23:52.113361 29711 ts_tablet_manager.cc:1939] T ec8426339bbd44439852c808da8dd26d P 448c674331194ea8afa98a5290d97334: Deleting consensus metadata
I20260929 10:23:52.113365 29713 ts_tablet_manager.cc:1939] T ec8426339bbd44439852c808da8dd26d P b2b327f9f45d42f19737f9ae6e3ca807: Deleting consensus metadata
I20260929 10:23:52.113658 29712 ts_tablet_manager.cc:1939] T ec8426339bbd44439852c808da8dd26d P 992dcee63bc743bb98db0cdb48aace54: Deleting consensus metadata
I20260929 10:23:52.113672 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 29250
I20260929 10:23:52.115236 29163 catalog_manager.cc:4841] TS 448c674331194ea8afa98a5290d97334 (127.21.33.131:38549): tablet ec8426339bbd44439852c808da8dd26d (table test-table [id=ede49ffa55f04b438351ae939939ee51]) successfully deleted
I20260929 10:23:52.115721 29163 catalog_manager.cc:4841] TS 992dcee63bc743bb98db0cdb48aace54 (127.21.33.130:36499): tablet ec8426339bbd44439852c808da8dd26d (table test-table [id=ede49ffa55f04b438351ae939939ee51]) successfully deleted
W20260929 10:23:52.121624 29163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:35021 (error 108)
I20260929 10:23:52.121640 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 29390
I20260929 10:23:52.123591 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:52.123103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:35021 (local address 127.0.0.1:53728)
0929 10:23:52.123371 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:52.123460 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
W20260929 10:23:52.123744 29163 catalog_manager.cc:4584] TS b2b327f9f45d42f19737f9ae6e3ca807 (127.21.33.129:35021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec8426339bbd44439852c808da8dd26d: Network error: Client connection negotiation failed: client connection to 127.21.33.129:35021: connect: Connection refused (error 111)
I20260929 10:23:52.128198 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 29530
I20260929 10:23:52.134260 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 29146
2026-09-29T10:23:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3161 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:52 dist-test-slave-2rbl krb5kdc[29782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:52 dist-test-slave-2rbl krb5kdc[29782](info): set up 2 sockets
Sep 29 10:23:52 dist-test-slave-2rbl krb5kdc[29782](info): commencing operation
krb5kdc: starting...
W20260929 10:23:53.186441 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:53 dist-test-slave-2rbl krb5kdc[29782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:53.312410 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:43995
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:43995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:53.421589 29795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:53.421898 29795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:53.421943 29795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:53.426218 29795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:53.426306 29795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:53.426328 29795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:53.426348 29795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:53.426409 29795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:53.431993 29795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:43995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:43995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:53.433238 29795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:53.434358 29795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:53.443284 29803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:53.443284 29800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:53.443450 29795 server_base.cc:1034] running on GCE node
W20260929 10:23:53.443277 29801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:53.444283 29795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:53.445500 29795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:53.446664 29795 hybrid_clock.cc:648] HybridClock initialized: now 1790677433446651 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T10:23:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1324 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:23:53 dist-test-slave-2rbl krb5kdc[29808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:23:53 dist-test-slave-2rbl krb5kdc[29808](info): set up 2 sockets
Sep 29 10:23:53 dist-test-slave-2rbl krb5kdc[29808](info): commencing operation
krb5kdc: starting...
W20260929 10:23:54.538450 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:23:54 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:23:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:23:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:23:54.664023 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46501
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:54.785810 29821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:54.786093 29821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:54.786178 29821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:54.790201 29821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:54.790285 29821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:54.790324 29821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:54.790382 29821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:54.790421 29821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:54.795580 29821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:54.796624 29821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:54.797784 29821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:54.806037 29827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:54.806046 29829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:54.806079 29821 server_base.cc:1034] running on GCE node
W20260929 10:23:54.806053 29826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:54.806782 29821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:54.807896 29821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:54.809082 29821 hybrid_clock.cc:648] HybridClock initialized: now 1790677434809068 us; error 34 us; skew 500 ppm
Sep 29 10:23:54 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677434, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:54.811735 29821 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:54.812613 29821 webserver.cc:460] Webserver started at http://127.21.33.190:37619/ using document root <none> and password file <none>
I20260929 10:23:54.813163 29821 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:54.813287 29821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:54.813521 29821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:54.815469 29821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "19a32a9e63794317993b080e9a8d1e6b"
format_stamp: "Formatted at 2026-09-29 10:23:54 on dist-test-slave-2rbl"
server_key: "ad3b2d8f941dcb979cbed2d75111d886"
server_key_iv: "6ea5279644606078567f9e691f9c8381"
server_key_version: "encryptionkey@0"
I20260929 10:23:54.816025 29821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "19a32a9e63794317993b080e9a8d1e6b"
format_stamp: "Formatted at 2026-09-29 10:23:54 on dist-test-slave-2rbl"
server_key: "ad3b2d8f941dcb979cbed2d75111d886"
server_key_iv: "6ea5279644606078567f9e691f9c8381"
server_key_version: "encryptionkey@0"
I20260929 10:23:54.819608 29821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:54.821925 29836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:54.823099 29821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:54.823336 29821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "19a32a9e63794317993b080e9a8d1e6b"
format_stamp: "Formatted at 2026-09-29 10:23:54 on dist-test-slave-2rbl"
server_key: "ad3b2d8f941dcb979cbed2d75111d886"
server_key_iv: "6ea5279644606078567f9e691f9c8381"
server_key_version: "encryptionkey@0"
I20260929 10:23:54.823450 29821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:54.837723 29821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:54.840404 29821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:54.840605 29821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:54.847606 29821 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:46501
I20260929 10:23:54.847633 29888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:46501 every 8 connection(s)
I20260929 10:23:54.848510 29821 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:54.849752 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 29821
I20260929 10:23:54.849895 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:23:54.850210 21638 external_mini_cluster.cc:1428] Setting key 871107a5be37e1bdb694f8fd7b3bf2ac
I20260929 10:23:54.851697 29889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:23:54.857010 29889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Bootstrap starting.
Sep 29 10:23:54 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:54.859411 29889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Neither blocks nor log segments found. Creating new log.
I20260929 10:23:54.860311 29889 log.cc:824] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:54.862221 29889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: No bootstrap required, opened a new log
I20260929 10:23:54.864037 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:54.851587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.21.33.190:46501)
0929 10:23:54.852021 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:54.852036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:54.852070 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:23:54.852686 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:54.852701 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:54.853012 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:54.853385 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:54.854167 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:54.855162 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:54.856028 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:54.856502 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:54.856671 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:54.859330 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:54.859336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:54.859348 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:54.859374 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:54.861345 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:54.861787 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:54.861798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:54.861803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:54.861910 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:54.862197 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:54.862200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:54.862202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:54.862636 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:54.862840 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:54.863095 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 10:23:54.863371 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":156,"threads_started":1}
I20260929 10:23:54.865756 29889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:54.865998 29889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:23:54.866101 29889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a32a9e63794317993b080e9a8d1e6b, State: Initialized, Role: FOLLOWER
I20260929 10:23:54.866508 29889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:54.866626 29889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:54.866673 29889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:54.866742 29889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:23:54.867517 29889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:54.867791 29889 leader_election.cc:302] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a32a9e63794317993b080e9a8d1e6b; no voters:
I20260929 10:23:54.868014 29889 leader_election.cc:288] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:23:54.868120 29894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:23:54.868378 29894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 LEADER]: Becoming Leader. State: Replica: 19a32a9e63794317993b080e9a8d1e6b, State: Running, Role: LEADER
I20260929 10:23:54.868733 29894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:54.869287 29889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:54.870402 29896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a32a9e63794317993b080e9a8d1e6b. Latest consensus state: current_term: 1 leader_uuid: "19a32a9e63794317993b080e9a8d1e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } } }
I20260929 10:23:54.870530 29896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:54.870932 29903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:54.871405 29895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a32a9e63794317993b080e9a8d1e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } } }
I20260929 10:23:54.871501 29895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:54.873823 29903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:54.878983 29903 catalog_manager.cc:1347] Generated new cluster ID: 75a1d1626da14f87aa293d315d0865ea
I20260929 10:23:54.879047 29903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:54.887238 29903 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:23:54.888092 29903 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:54.912029 29903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Generated new TSK 0
I20260929 10:23:54.912813 29903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:23:54.951812 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46501
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:55.062783 29915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:55.063053 29915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:55.063097 29915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:55.067214 29915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:55.067265 29915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:55.067350 29915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:55.072719 29915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:55.073796 29915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:55.074801 29915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:55.086750 29920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.086783 29923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.086850 29921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:55.087105 29915 server_base.cc:1034] running on GCE node
I20260929 10:23:55.087530 29915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:55.088224 29915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:55.089422 29915 hybrid_clock.cc:648] HybridClock initialized: now 1790677435089401 us; error 31 us; skew 500 ppm
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:55.092217 29915 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:23:55.093103 29915 webserver.cc:460] Webserver started at http://127.21.33.129:39663/ using document root <none> and password file <none>
I20260929 10:23:55.093700 29915 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:55.093794 29915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:55.094025 29915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:55.096064 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "c2e175fd02f3469096325ed68ce0e648"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "ded1680dc21dd24f7887bc5019a562db"
server_key_iv: "451e780895decb2c314874599fb8d2f2"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.096638 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "c2e175fd02f3469096325ed68ce0e648"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "ded1680dc21dd24f7887bc5019a562db"
server_key_iv: "451e780895decb2c314874599fb8d2f2"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.100443 29915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:23:55.102846 29930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.103983 29915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:23:55.104131 29915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "c2e175fd02f3469096325ed68ce0e648"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "ded1680dc21dd24f7887bc5019a562db"
server_key_iv: "451e780895decb2c314874599fb8d2f2"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.104250 29915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:55.126616 29915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:55.129791 29915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:55.129998 29915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:55.130613 29915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:55.131445 29915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:55.131517 29915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.131587 29915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:55.131623 29915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.140461 29915 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:39491
I20260929 10:23:55.140488 30043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:39491 every 8 connection(s)
I20260929 10:23:55.141435 29915 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:55.148353 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 29915
I20260929 10:23:55.148458 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:23:55.148765 21638 external_mini_cluster.cc:1428] Setting key f4fb4227e837f86552ad967a338f48f1
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:55.156531 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:55.143257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:40603 (local address 127.21.33.190:46501)
0929 10:23:55.143442 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:55.143446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:55.144317 (+ 871us) server_negotiation.cc:407] Connection header received
0929 10:23:55.145596 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:55.145599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:55.145647 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:55.145721 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:55.146890 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.147474 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.148795 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.149176 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.149339 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:55.152380 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:55.152384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:55.152386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:55.152409 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:55.154223 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:55.154869 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:55.154875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:55.154880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:55.154983 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:55.155371 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:55.155374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:55.155375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:55.155549 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:55.155624 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:55.156300 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 10:23:55.156406 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 10:23:55.157788 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:55.143683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:40603)
0929 10:23:55.144156 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:55.144202 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:55.145392 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:55.145863 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:55.145870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:55.146235 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:55.146740 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:55.146755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.147626 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.147629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:55.148659 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:55.148670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.149395 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.149399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:55.149598 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:55.150312 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:55.150334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:55.152178 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:55.154343 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:55.154352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:55.154370 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:55.154766 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:55.155061 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:55.155066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:55.155069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:55.155270 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:55.155658 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:55.155667 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:55.156019 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 10:23:55.156272 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 10:23:55.156658 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
I20260929 10:23:55.159111 30044 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46501
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:23:55.159390 30044 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:55.159900 30044 heartbeater.cc:511] Master 127.21.33.190:46501 requested a full tablet report, sending...
I20260929 10:23:55.161525 29853 ts_manager.cc:194] Registered new tserver with Master: c2e175fd02f3469096325ed68ce0e648 (127.21.33.129:39491)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:23:55.163090 29853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:40603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:23:55.181027 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46501
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:55.290091 30049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:55.290331 30049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:55.290405 30049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:55.294437 30049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:55.294492 30049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:55.294602 30049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:55.300000 30049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:55.301015 30049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:55.302070 30049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:55.311967 30057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.312033 30055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.312068 30054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:55.312153 30049 server_base.cc:1034] running on GCE node
I20260929 10:23:55.312685 30049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:55.313349 30049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:55.314502 30049 hybrid_clock.cc:648] HybridClock initialized: now 1790677435314469 us; error 46 us; skew 500 ppm
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:55.317147 30049 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:23:55.318032 30049 webserver.cc:460] Webserver started at http://127.21.33.130:38973/ using document root <none> and password file <none>
I20260929 10:23:55.318559 30049 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:55.318625 30049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:55.318789 30049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:55.320708 30049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "791809e6d0f24d4d9111444c553f7fb3"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "1bc37b95cef9d470fda6bb6d98b6f309"
server_key_iv: "622596fccb63ef366577ac6145884051"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.321274 30049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "791809e6d0f24d4d9111444c553f7fb3"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "1bc37b95cef9d470fda6bb6d98b6f309"
server_key_iv: "622596fccb63ef366577ac6145884051"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.324708 30049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 10:23:55.327188 30064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.328570 30049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:23:55.328703 30049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "791809e6d0f24d4d9111444c553f7fb3"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "1bc37b95cef9d470fda6bb6d98b6f309"
server_key_iv: "622596fccb63ef366577ac6145884051"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.328812 30049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:55.340883 30049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:55.343779 30049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:55.343976 30049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:55.344542 30049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:55.345388 30049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:55.345455 30049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.345521 30049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:55.345568 30049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.354756 30049 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:40353
I20260929 10:23:55.354791 30177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:40353 every 8 connection(s)
I20260929 10:23:55.355623 30049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:55.357288 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30049
I20260929 10:23:55.357450 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:23:55.357725 21638 external_mini_cluster.cc:1428] Setting key 31e951bfe4d3fe5ad78c9147b29cd923
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:23:55.370723 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:55.357351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:49195 (local address 127.21.33.190:46501)
0929 10:23:55.357488 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:55.357493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:55.358454 (+ 961us) server_negotiation.cc:407] Connection header received
0929 10:23:55.359817 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:55.359824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:55.359888 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:55.359987 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:55.361460 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.362000 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.363154 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.363482 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.363613 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:55.366844 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:55.366848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:55.366852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:55.366881 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:55.368514 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:55.368988 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:55.368991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:55.368992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:55.369098 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:55.369549 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:55.369552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:55.369553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:55.369728 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:55.369807 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:55.370445 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 10:23:55.370588 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:23:55.371940 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:55.357757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:49195)
0929 10:23:55.358259 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:55.358323 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:55.359602 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:55.360143 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:55.360149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:55.360505 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:55.361287 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:55.361310 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.362127 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.362131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:55.363020 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:55.363029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.363671 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.363675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:55.363919 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:55.364651 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:55.364675 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:55.366654 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:55.368617 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:55.368622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:55.368634 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:55.368887 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:55.369299 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:55.369304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:55.369311 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:55.369449 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:55.369824 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:55.369834 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:55.370161 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 10:23:55.370359 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 10:23:55.370773 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
I20260929 10:23:55.373162 30178 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46501
I20260929 10:23:55.373431 30178 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:55.373907 30178 heartbeater.cc:511] Master 127.21.33.190:46501 requested a full tablet report, sending...
I20260929 10:23:55.374935 29853 ts_manager.cc:194] Registered new tserver with Master: 791809e6d0f24d4d9111444c553f7fb3 (127.21.33.130:40353)
I20260929 10:23:55.375496 29853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:49195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:23:55.388603 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46501
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:55.497690 30183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:55.497963 30183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:55.498034 30183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:55.502220 30183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:55.502305 30183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:55.502393 30183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:55.507686 30183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:55.508760 30183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:55.509809 30183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:55.519676 30189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.519781 30191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:55.519840 30183 server_base.cc:1034] running on GCE node
W20260929 10:23:55.519696 30188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:55.520258 30183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:55.520862 30183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:55.522004 30183 hybrid_clock.cc:648] HybridClock initialized: now 1790677435521972 us; error 41 us; skew 500 ppm
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:55.524616 30183 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:23:55.525477 30183 webserver.cc:460] Webserver started at http://127.21.33.131:34569/ using document root <none> and password file <none>
I20260929 10:23:55.526011 30183 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:55.526069 30183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:55.526314 30183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:23:55.528366 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "b530705d832441fdad49ae35d01350a2"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "e670661b1648da5662d4e2ef0b43d123"
server_key_iv: "17a4f0d1a3e7fc5668c3276412a66ad0"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.528909 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "b530705d832441fdad49ae35d01350a2"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "e670661b1648da5662d4e2ef0b43d123"
server_key_iv: "17a4f0d1a3e7fc5668c3276412a66ad0"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.532490 30183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:23:55.534819 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.535959 30183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 10:23:55.536082 30183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "b530705d832441fdad49ae35d01350a2"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "e670661b1648da5662d4e2ef0b43d123"
server_key_iv: "17a4f0d1a3e7fc5668c3276412a66ad0"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.536180 30183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:55.544600 30183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:55.547235 30183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:55.547422 30183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:55.547966 30183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:55.548724 30183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:55.548780 30183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.548844 30183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:55.548883 30183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.557070 30183 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:38971
I20260929 10:23:55.557093 30311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:38971 every 8 connection(s)
I20260929 10:23:55.557934 30183 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:23:55.564487 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30183
I20260929 10:23:55.564625 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:23:55.564919 21638 external_mini_cluster.cc:1428] Setting key cc5a4c313c62f07c48fec8c52169fb09
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:23:55.571026 29892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:55.559644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:49431 (local address 127.21.33.190:46501)
0929 10:23:55.559779 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:55.559782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:55.560535 (+ 753us) server_negotiation.cc:407] Connection header received
0929 10:23:55.561762 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:55.561764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:55.561813 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:55.561888 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:55.562953 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.563476 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.564779 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.564990 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.565110 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:55.567811 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:55.567814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:55.567816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:55.567838 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:55.569090 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:55.569591 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:55.569594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:55.569595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:55.569668 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:55.569994 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:55.569996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:55.569997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:55.570178 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:55.570261 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:55.570744 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 10:23:55.570864 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:23:55.571785 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:55.559935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:49431)
0929 10:23:55.560381 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:55.560421 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:55.561562 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:55.562036 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:55.562042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:55.562379 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:55.562823 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:55.562835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.563576 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.563578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:55.564626 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:55.564638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:55.565167 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:55.565172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:55.565391 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:55.565978 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:55.565993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:55.567671 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:23:55.569183 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:55.569187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:55.569235 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:55.569484 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:55.569743 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:23:55.569745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:23:55.569747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:23:55.569875 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:23:55.570260 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:23:55.570264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:23:55.570469 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 10:23:55.570674 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 10:23:55.570986 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":96,"threads_started":1}
I20260929 10:23:55.572772 30312 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46501
I20260929 10:23:55.573004 30312 heartbeater.cc:464] Registering TS with master...
I20260929 10:23:55.573500 30312 heartbeater.cc:511] Master 127.21.33.190:46501 requested a full tablet report, sending...
I20260929 10:23:55.574436 29853 ts_manager.cc:194] Registered new tserver with Master: b530705d832441fdad49ae35d01350a2 (127.21.33.131:38971)
I20260929 10:23:55.574975 29853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:49431
I20260929 10:23:55.578593 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab.
Entry for principal oryx/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab.
WARNING: no policy specified for oryx/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab.
Entry for principal oryx/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab.
WARNING: no policy specified for oryx/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab.
Entry for principal oryx/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab.
I20260929 10:23:55.647799 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 29915
I20260929 10:23:55.653569 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30049
I20260929 10:23:55.658624 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30183
I20260929 10:23:55.663568 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 29821
I20260929 10:23:55.669628 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46501
--webserver_interface=127.21.33.190
--webserver_port=37619
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:23:55.779184 30321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:55.779470 30321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:55.779551 30321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:55.783488 30321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:23:55.783573 30321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:55.783613 30321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:55.783670 30321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:23:55.783723 30321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:23:55.788932 30321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:46501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=37619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:55.790015 30321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:55.791028 30321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:55.799420 30327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.799499 30326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.799739 30329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:55.799554 30321 server_base.cc:1034] running on GCE node
I20260929 10:23:55.800334 30321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:55.801455 30321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:55.802644 30321 hybrid_clock.cc:648] HybridClock initialized: now 1790677435802623 us; error 32 us; skew 500 ppm
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:55.805375 30321 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:23:55.806233 30321 webserver.cc:460] Webserver started at http://127.21.33.190:37619/ using document root <none> and password file <none>
I20260929 10:23:55.806790 30321 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:55.806897 30321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:55.810632 30321 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 10:23:55.812595 30336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:55.813853 30321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:23:55.813992 30321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "19a32a9e63794317993b080e9a8d1e6b"
format_stamp: "Formatted at 2026-09-29 10:23:54 on dist-test-slave-2rbl"
server_key: "ad3b2d8f941dcb979cbed2d75111d886"
server_key_iv: "6ea5279644606078567f9e691f9c8381"
server_key_version: "encryptionkey@0"
I20260929 10:23:55.814522 30321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:55.828449 30321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:55.831255 30321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:55.831465 30321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:55.838723 30321 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:46501
I20260929 10:23:55.838739 30388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:46501 every 8 connection(s)
I20260929 10:23:55.839811 30321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:23:55.843966 30389 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:23:55.845695 30389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Bootstrap starting.
I20260929 10:23:55.845727 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30321
I20260929 10:23:55.846155 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:39491
--local_ip_for_outbound_sockets=127.21.33.129
--tserver_master_addrs=127.21.33.190:46501
--webserver_port=39663
--webserver_interface=127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:23:55.868253 30389 log.cc:824] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:23:55.876358 30389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:23:55.876781 30389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Bootstrap complete.
I20260929 10:23:55.882441 30389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:55.882948 30389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a32a9e63794317993b080e9a8d1e6b, State: Initialized, Role: FOLLOWER
I20260929 10:23:55.883481 30389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:55.883620 30389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:23:55.883690 30389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:23:55.883790 30389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:23:55.886246 30389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:55.886593 30389 leader_election.cc:302] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a32a9e63794317993b080e9a8d1e6b; no voters:
I20260929 10:23:55.886927 30389 leader_election.cc:288] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:23:55.887094 30393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:23:55.887359 30393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [term 2 LEADER]: Becoming Leader. State: Replica: 19a32a9e63794317993b080e9a8d1e6b, State: Running, Role: LEADER
I20260929 10:23:55.887669 30393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } }
I20260929 10:23:55.888185 30389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:23:55.889235 30395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a32a9e63794317993b080e9a8d1e6b. Latest consensus state: current_term: 2 leader_uuid: "19a32a9e63794317993b080e9a8d1e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } } }
I20260929 10:23:55.889360 30395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:55.889206 30394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "19a32a9e63794317993b080e9a8d1e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a32a9e63794317993b080e9a8d1e6b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46501 } } }
I20260929 10:23:55.889459 30394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b [sys.catalog]: This master's current role is: LEADER
I20260929 10:23:55.893478 30408 catalog_manager.cc:1259] Loaded cluster ID: 75a1d1626da14f87aa293d315d0865ea
I20260929 10:23:55.893538 30408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: loading cluster ID for follower catalog manager: success
I20260929 10:23:55.895908 30408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: acquiring CA information for follower catalog manager: success
I20260929 10:23:55.897287 30408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:23:55.897404 30402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:23:55.898247 30402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:23:55.898692 30402 catalog_manager.cc:1259] Loaded cluster ID: 75a1d1626da14f87aa293d315d0865ea
I20260929 10:23:55.898761 30402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:23:55.899516 30402 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:23:55.899981 30402 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19a32a9e63794317993b080e9a8d1e6b: Loaded TSK: 0
I20260929 10:23:55.900784 30402 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:23:55.972167 30391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:55.972486 30391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:55.972570 30391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:55.976713 30391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:55.976809 30391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:55.976903 30391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:23:55.982307 30391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.129.keytab
--principal=oryx/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=39663
--tserver_master_addrs=127.21.33.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:55.983315 30391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:55.984325 30391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:55.994139 30416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.994181 30415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:55.994171 30418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:55.994405 30391 server_base.cc:1034] running on GCE node
I20260929 10:23:55.994971 30391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:55.995769 30391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:55.996979 30391 hybrid_clock.cc:648] HybridClock initialized: now 1790677435996950 us; error 48 us; skew 500 ppm
Sep 29 10:23:55 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677435, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:56.000519 30391 init.cc:377] Logged in from keytab as oryx/127.21.33.129@KRBTEST.COM (short username oryx)
I20260929 10:23:56.001649 30391 webserver.cc:460] Webserver started at http://127.21.33.129:39663/ using document root <none> and password file <none>
I20260929 10:23:56.002379 30391 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:56.002496 30391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:56.007503 30391 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:23:56.010118 30425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.011628 30391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:23:56.011761 30391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "c2e175fd02f3469096325ed68ce0e648"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "ded1680dc21dd24f7887bc5019a562db"
server_key_iv: "451e780895decb2c314874599fb8d2f2"
server_key_version: "encryptionkey@0"
I20260929 10:23:56.012456 30391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:56.051352 30391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:56.054293 30391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:56.054504 30391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:56.055100 30391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:56.055907 30391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:56.055953 30391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.056020 30391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:56.056051 30391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.064757 30391 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:39491
I20260929 10:23:56.064832 30538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:39491 every 8 connection(s)
I20260929 10:23:56.065812 30391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:23:56.073827 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30391
I20260929 10:23:56.074306 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:40353
--local_ip_for_outbound_sockets=127.21.33.130
--tserver_master_addrs=127.21.33.190:46501
--webserver_port=38973
--webserver_interface=127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.078508 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.067853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:36013)
0929 10:23:56.068284 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.068334 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.069575 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.070579 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.070585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.070939 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.071370 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.071383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.072567 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.072569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.073490 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.073500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.074293 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.074298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.074515 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.075181 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.075208 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.077714 (+ 2506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":131,"threads_started":1}
W20260929 10:23:56.078969 30539 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46501 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.33.190:46501: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
W20260929 10:23:56.079020 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.067790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:36013 (local address 127.21.33.190:46501)
0929 10:23:56.068217 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.068236 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.068441 (+ 205us) server_negotiation.cc:407] Connection header received
0929 10:23:56.069929 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.069940 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.070227 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.070498 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.071519 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.072451 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.073606 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.074135 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.074257 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.078037 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:36013: BlockingRecv error: recv got EOF from 127.21.33.129:36013 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.084920 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.079610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:57431)
0929 10:23:56.079722 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.079736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.079807 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.080047 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.080050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.080224 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.080311 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.080316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.081036 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.081039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.081645 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.081654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.082109 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.082113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.082209 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.082751 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.082765 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.084784 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 10:23:56.085012 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.079715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:57431 (local address 127.21.33.190:46501)
0929 10:23:56.079844 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.079850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.079865 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:23:56.079913 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.079920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.079955 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.080034 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.080421 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.080921 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.081776 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.081993 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.082097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.084906 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:57431: BlockingRecv error: recv got EOF from 127.21.33.129:57431 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.090415 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.085573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:35155)
0929 10:23:56.085658 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.085671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.085765 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.086011 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.086013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.086158 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.086221 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.086226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.086909 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.086915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.087423 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.087430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.087898 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.087901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.087986 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.088458 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.088470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.090254 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 10:23:56.090512 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.085676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:35155 (local address 127.21.33.190:46501)
0929 10:23:56.085824 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.085830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.085843 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:56.085873 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.085875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.085917 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.085997 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.086328 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.086806 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.087537 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.087785 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.087887 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.090381 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:35155: BlockingRecv error: recv got EOF from 127.21.33.129:35155 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 10:23:56.090746 30539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:23:56.186182 30543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:56.186420 30543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:56.186461 30543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:56.190894 30543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:56.190949 30543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:56.191046 30543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:23:56.196188 30543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.130.keytab
--principal=oryx/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:40353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=38973
--tserver_master_addrs=127.21.33.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:56.197160 30543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:56.198185 30543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:23:56.207510 30549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:56.207562 30543 server_base.cc:1034] running on GCE node
W20260929 10:23:56.207662 30548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:56.207695 30551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:56.208105 30543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:56.208724 30543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:56.209903 30543 hybrid_clock.cc:648] HybridClock initialized: now 1790677436209874 us; error 43 us; skew 500 ppm
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677436, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:56.212460 30543 init.cc:377] Logged in from keytab as oryx/127.21.33.130@KRBTEST.COM (short username oryx)
I20260929 10:23:56.213335 30543 webserver.cc:460] Webserver started at http://127.21.33.130:38973/ using document root <none> and password file <none>
I20260929 10:23:56.213884 30543 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:56.213989 30543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:56.218034 30543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 10:23:56.220064 30558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.221387 30543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 10:23:56.221474 30543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "791809e6d0f24d4d9111444c553f7fb3"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "1bc37b95cef9d470fda6bb6d98b6f309"
server_key_iv: "622596fccb63ef366577ac6145884051"
server_key_version: "encryptionkey@0"
I20260929 10:23:56.221974 30543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:56.240967 30543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:56.244400 30543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:56.244557 30543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:56.245107 30543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:56.245970 30543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:56.246016 30543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.246057 30543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:56.246073 30543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.254554 30543 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:40353
I20260929 10:23:56.254624 30671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:40353 every 8 connection(s)
I20260929 10:23:56.255573 30543 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:23:56.261199 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30543
I20260929 10:23:56.261698 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:38971
--local_ip_for_outbound_sockets=127.21.33.131
--tserver_master_addrs=127.21.33.190:46501
--webserver_port=34569
--webserver_interface=127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 10:23:56.266978 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.257406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:60051 (local address 127.21.33.190:46501)
0929 10:23:56.257570 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.257575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.258451 (+ 876us) server_negotiation.cc:407] Connection header received
0929 10:23:56.259660 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.259664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.259714 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.259791 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.261074 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.261899 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.263028 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.263264 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.263376 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.266857 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:60051: BlockingRecv error: recv got EOF from 127.21.33.130:60051 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:23:56.267638 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.257855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:60051)
0929 10:23:56.258294 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.258347 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.259471 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.259957 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.259964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.260378 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.260850 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.260863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.262018 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.262021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.262897 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.262905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.263412 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.263416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.263590 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.264203 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.264218 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.266725 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
W20260929 10:23:56.268193 30672 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46501 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.33.190:46501: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.273638 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.268668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:39159)
0929 10:23:56.268792 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.268808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.268883 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.269107 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.269110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.269302 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.269387 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.269393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.270090 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.270097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.270653 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.270660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.271105 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.271108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.271196 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.271712 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.271723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.273502 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 10:23:56.273736 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.268779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:39159 (local address 127.21.33.190:46501)
0929 10:23:56.268896 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.268901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.268919 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:23:56.268962 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.268965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.269010 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.269094 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.269499 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.269985 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.270783 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.270997 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.271104 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.273618 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:39159: BlockingRecv error: recv got EOF from 127.21.33.130:39159 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.279265 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.274284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:60307)
0929 10:23:56.274408 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.274422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.274494 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.274710 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.274713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.274874 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.274958 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.274963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.275605 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.275608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.276469 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.276478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.276863 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.276866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.276947 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.277427 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.277441 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.279131 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 10:23:56.279451 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.274337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:60307 (local address 127.21.33.190:46501)
0929 10:23:56.274452 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.274455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.274467 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:23:56.274578 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.274581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.274615 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.274676 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.275056 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.275513 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.276575 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.276767 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.276838 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.279244 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:60307: BlockingRecv error: recv got EOF from 127.21.33.130:60307 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 10:23:56.279450 30672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:23:56.374239 30675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:23:56.374540 30675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:23:56.374615 30675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:23:56.378687 30675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:23:56.378775 30675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:23:56.378868 30675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:23:56.384214 30675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/krb5kdc/oryx_127.21.33.131.keytab
--principal=oryx/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:38971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=34569
--tserver_master_addrs=127.21.33.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:23:56.385345 30675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:23:56.386397 30675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:23:56.396631 30675 server_base.cc:1034] running on GCE node
W20260929 10:23:56.396589 30680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:56.396582 30681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:23:56.396626 30683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:23:56.397333 30675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:23:56.398000 30675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:23:56.399190 30675 hybrid_clock.cc:648] HybridClock initialized: now 1790677436399145 us; error 60 us; skew 500 ppm
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677436, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:23:56.402172 30675 init.cc:377] Logged in from keytab as oryx/127.21.33.131@KRBTEST.COM (short username oryx)
I20260929 10:23:56.403020 30675 webserver.cc:460] Webserver started at http://127.21.33.131:34569/ using document root <none> and password file <none>
I20260929 10:23:56.403566 30675 fs_manager.cc:359] Metadata directory not provided
I20260929 10:23:56.403640 30675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:23:56.407922 30675 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:23:56.410169 30690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.411355 30675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:23:56.411485 30675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "b530705d832441fdad49ae35d01350a2"
format_stamp: "Formatted at 2026-09-29 10:23:55 on dist-test-slave-2rbl"
server_key: "e670661b1648da5662d4e2ef0b43d123"
server_key_iv: "17a4f0d1a3e7fc5668c3276412a66ad0"
server_key_version: "encryptionkey@0"
I20260929 10:23:56.412035 30675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:23:56.433109 30675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:23:56.436234 30675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:23:56.436458 30675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:23:56.437101 30675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:23:56.438027 30675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:23:56.438091 30675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.438160 30675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:23:56.438196 30675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:23:56.447430 30675 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:38971
I20260929 10:23:56.447448 30803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:38971 every 8 connection(s)
I20260929 10:23:56.448513 30675 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.458586 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30675
W20260929 10:23:56.459236 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.450342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:53117 (local address 127.21.33.190:46501)
0929 10:23:56.450518 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.450523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.451333 (+ 810us) server_negotiation.cc:407] Connection header received
0929 10:23:56.452533 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.452535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.452583 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.452657 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.453784 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.454309 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.455364 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.455567 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.455642 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.459091 (+ 3449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:53117: BlockingRecv error: recv got EOF from 127.21.33.131:53117 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:23:56.459915 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.450754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:53117)
0929 10:23:56.451174 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.451217 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.452351 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.452812 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.452818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.453155 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.453661 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.453674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.454412 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.454415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.455254 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.455261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.455686 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.455688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.455820 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.456425 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.456441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.458939 (+ 2498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
W20260929 10:23:56.460402 30804 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46501 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.33.190:46501: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.466372 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.460851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:41461)
0929 10:23:56.460950 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.460964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.461034 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.461387 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.461389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.461542 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.461638 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.461643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.462658 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.462660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.463426 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.463435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.463886 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.463890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.463983 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.464420 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.464430 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.466223 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 10:23:56.466470 30808 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.460914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:41461 (local address 127.21.33.190:46501)
0929 10:23:56.461150 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.461153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.461164 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:23:56.461231 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.461234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.461273 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.461351 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.461756 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.462547 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.463540 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.463779 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.463883 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.466338 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:41461: BlockingRecv error: recv got EOF from 127.21.33.131:41461 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":112,"threads_started":1}
I20260929 10:23:56.468885 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.459623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.21.33.190:46501)
0929 10:23:56.459731 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.459735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.459749 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:56.459790 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.459793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.459833 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.459918 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.460838 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.461642 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.462814 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.463136 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.463238 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.464761 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:23:56.464771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:23:56.464794 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:23:56.464832 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:23:56.466738 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:56.467213 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:56.467218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:56.467223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:56.467415 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:23:56.467707 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:23:56.467709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:23:56.467710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:23:56.468044 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:23:56.468188 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 10:23:56.468517 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 10:23:56.468745 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:56 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:56.471938 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:56.466898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:44773)
0929 10:23:56.466984 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:56.466998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:56.467066 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:56.467311 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:56.467314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:56.467462 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:56.467549 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.467555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.468293 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.468295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.469053 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:56.469071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.469546 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.469549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:56.469626 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.470025 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:56.470039 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:56.471858 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 10:23:56.472119 30804 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:23:56.472307 30808 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:56.466953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:44773 (local address 127.21.33.190:46501)
0929 10:23:56.467095 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:56.467098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:56.467111 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:23:56.467145 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:56.467148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:56.467195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:56.467274 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:56.467662 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.468179 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.469177 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:56.469435 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:56.469545 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:56.471989 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:44773: BlockingRecv error: recv got EOF from 127.21.33.131:44773 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 10:23:57 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:57 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:57.096546 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:57.091384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:36695)
0929 10:23:57.091652 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:57.091670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:57.091790 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:57.092029 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:57.092031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:57.092232 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:57.092355 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:57.092362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.093073 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.093076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:57.093679 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:57.093688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.094119 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.094121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:57.094192 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:57.094670 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:57.094682 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:57.096431 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
W20260929 10:23:57.096689 30810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:57.091421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:36695 (local address 127.21.33.190:46501)
0929 10:23:57.091662 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:57.091666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:57.091754 (+ 88us) server_negotiation.cc:407] Connection header received
0929 10:23:57.091870 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:57.091872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:57.091915 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:57.091994 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:57.092456 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.092971 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.093800 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.094014 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.094087 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:57.096554 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:36695: BlockingRecv error: recv got EOF from 127.21.33.129:36695 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
Sep 29 10:23:57 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:57 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:57.285544 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:57.280011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:58199)
0929 10:23:57.280352 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:57.280370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:57.280522 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:57.280800 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:57.280803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:57.281008 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:57.281122 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:57.281127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.281907 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.281913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:57.282498 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:57.282506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.282960 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.282964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:57.283058 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:57.283539 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:57.283550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:57.285379 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
W20260929 10:23:57.285604 30810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:57.280202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:58199 (local address 127.21.33.190:46501)
0929 10:23:57.280354 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:57.280359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:57.280460 (+ 101us) server_negotiation.cc:407] Connection header received
0929 10:23:57.280625 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:57.280630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:57.280685 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:57.280794 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:57.281255 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.281794 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.282619 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.282847 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.282958 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:57.285498 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:58199: BlockingRecv error: recv got EOF from 127.21.33.130:58199 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 10:23:57 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:57 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:57.478456 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:57.473021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:54421)
0929 10:23:57.473348 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:57.473361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:57.473459 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:57.473692 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:57.473694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:57.473883 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:57.473996 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:57.474002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.474720 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.474722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:57.475273 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:57.475280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.475695 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.475698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:57.475777 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:57.476313 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:57.476333 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:57.478326 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":93,"threads_started":1}
W20260929 10:23:57.478576 30810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:57.473080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:54421 (local address 127.21.33.190:46501)
0929 10:23:57.473269 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:57.473275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:57.473438 (+ 163us) server_negotiation.cc:407] Connection header received
0929 10:23:57.473541 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:57.473543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:57.473589 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:57.473658 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:57.474095 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.474630 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.475378 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:57.475591 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:57.475665 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:57.478459 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:54421: BlockingRecv error: recv got EOF from 127.21.33.131:54421 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 10:23:58 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:58 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:58.102872 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:58.097539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:55489)
0929 10:23:58.097848 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:58.097861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:58.097962 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:58.098202 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:58.098204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:58.098393 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:58.098483 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:58.098488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.099181 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.099183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:58.099714 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:58.099721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.100129 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.100132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:58.100193 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:58.100665 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:58.100675 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:58.102680 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":79,"threads_started":1}
W20260929 10:23:58.102931 30814 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:58.097596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:55489 (local address 127.21.33.190:46501)
0929 10:23:58.097883 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:58.097886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:58.097914 (+ 28us) server_negotiation.cc:407] Connection header received
0929 10:23:58.098044 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:58.098047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:58.098088 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:58.098163 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:58.098580 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.099092 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.099818 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.100026 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.100099 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:58.102808 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:55489: BlockingRecv error: recv got EOF from 127.21.33.129:55489 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":116,"threads_started":1}
Sep 29 10:23:58 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:58 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:58.291975 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:58.286360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:54383)
0929 10:23:58.286622 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:58.286636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:58.286724 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:58.286977 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:58.286979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:58.287146 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:58.287239 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:58.287244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.288177 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.288180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:58.289046 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:58.289055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.289547 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.289550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:58.289608 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:58.290117 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:58.290128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:58.291868 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
W20260929 10:23:58.292111 30814 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:58.286482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:54383 (local address 127.21.33.190:46501)
0929 10:23:58.286765 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:58.286769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:58.286783 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:58.286821 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:58.286827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:58.286868 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:58.286944 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:58.287372 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.288045 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.289143 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.289454 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.289536 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:58.291995 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:54383: BlockingRecv error: recv got EOF from 127.21.33.130:54383 (error 108)
Metrics: {"server-negotiator.queue_time_us":193}
Sep 29 10:23:58 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:58 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:58.484161 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:58.479163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:59249)
0929 10:23:58.479535 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:58.479547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:58.479638 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:58.479854 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:58.479856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:58.480026 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:58.480121 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:58.480127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.480820 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.480822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:58.481458 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:58.481467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.481885 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.481888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:58.481945 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:58.482389 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:58.482399 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:58.484047 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":205,"threads_started":1}
W20260929 10:23:58.484304 30814 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:58.479304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:59249 (local address 127.21.33.190:46501)
0929 10:23:58.479437 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:58.479441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:58.479609 (+ 168us) server_negotiation.cc:407] Connection header received
0929 10:23:58.479702 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:58.479704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:58.479752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:58.479819 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:58.480214 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.480728 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.481559 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:58.481778 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:58.481856 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:58.484175 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:59249: BlockingRecv error: recv got EOF from 127.21.33.131:59249 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 10:23:59 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:59 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:59.109355 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:59.103672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:36483)
0929 10:23:59.103960 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:59.103974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:59.104076 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:59.104351 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:59.104354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:59.104535 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:59.104637 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:59.104643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.105398 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.105401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:59.106102 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:59.106111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.106680 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.106683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:59.106780 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:59.107422 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:59.107436 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:59.109233 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
W20260929 10:23:59.109512 30818 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:59.103771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:36483 (local address 127.21.33.190:46501)
0929 10:23:59.103993 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:59.103996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:59.104023 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:23:59.104158 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:59.104161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:59.104202 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:59.104289 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:59.104745 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.105289 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.106227 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.106545 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.106649 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:59.109382 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:36483: BlockingRecv error: recv got EOF from 127.21.33.129:36483 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
Sep 29 10:23:59 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:59 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:59.298416 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:59.292773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:44237)
0929 10:23:59.293071 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:59.293093 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:59.293242 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:59.293487 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:59.293489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:59.293670 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:59.293764 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:59.293769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.294460 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.294462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:59.295028 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:59.295035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.295427 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.295429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:59.295489 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:59.296126 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:59.296144 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:59.298281 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
W20260929 10:23:59.298564 30818 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:59.292849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:44237 (local address 127.21.33.190:46501)
0929 10:23:59.292995 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:59.293001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:59.293166 (+ 165us) server_negotiation.cc:407] Connection header received
0929 10:23:59.293334 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:59.293337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:59.293380 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:59.293444 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:59.293861 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.294370 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.295132 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.295339 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.295407 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:59.298439 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:44237: BlockingRecv error: recv got EOF from 127.21.33.130:44237 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 10:23:59 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:23:59 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:23:59.490111 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:23:59.484963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:33893)
0929 10:23:59.485236 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 10:23:59.485249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:23:59.485341 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:23:59.485565 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:23:59.485566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:23:59.485737 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:23:59.485830 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:59.485835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.486623 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.486625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:59.487164 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:23:59.487171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.487621 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.487624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:23:59.487682 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:59.488157 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:23:59.488167 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:23:59.490002 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
W20260929 10:23:59.490296 30818 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:23:59.485087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:33893 (local address 127.21.33.190:46501)
0929 10:23:59.485295 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 10:23:59.485298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:23:59.485312 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:23:59.485422 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:23:59.485425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:23:59.485468 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:23:59.485533 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:23:59.485959 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.486503 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.487289 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:23:59.487518 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:23:59.487636 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:23:59.490131 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:33893: BlockingRecv error: recv got EOF from 127.21.33.131:33893 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 10:24:00 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:00 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:00.119944 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:00.110114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:43077)
0929 10:24:00.110388 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:00.110401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:00.110495 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:00.110806 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:00.110808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:00.110979 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:00.111068 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:00.111073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.111831 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.111833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:00.112369 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:00.112375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.112838 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.112840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:00.112897 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:00.117857 (+ 4960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:00.117870 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:00.119828 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
W20260929 10:24:00.120143 30822 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:00.110161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:43077 (local address 127.21.33.190:46501)
0929 10:24:00.110401 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:00.110405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:00.110466 (+ 61us) server_negotiation.cc:407] Connection header received
0929 10:24:00.110570 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:00.110572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:00.110664 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:00.110758 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:00.111189 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.111738 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.112476 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.112740 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.112821 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:00.119977 (+ 7156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:43077: BlockingRecv error: recv got EOF from 127.21.33.129:43077 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":60,"threads_started":1}
Sep 29 10:24:00 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:00 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:00.304652 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:00.299301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:46399)
0929 10:24:00.299557 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:00.299570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:00.299674 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:00.299908 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:00.299910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:00.300082 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:00.300182 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:00.300187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.300935 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.300937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:00.301566 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:00.301575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.302090 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.302094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:00.302180 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:00.302738 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:00.302754 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:00.304532 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
W20260929 10:24:00.304818 30822 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:00.299375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:46399 (local address 127.21.33.190:46501)
0929 10:24:00.299543 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:00.299547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:00.299646 (+ 99us) server_negotiation.cc:407] Connection header received
0929 10:24:00.299753 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:00.299755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:00.299797 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:00.299874 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:00.300281 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.300829 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.301713 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.301985 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.302086 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:00.304667 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:46399: BlockingRecv error: recv got EOF from 127.21.33.130:46399 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 10:24:00 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:00 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:00.495877 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:00.490882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:43289)
0929 10:24:00.491163 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:00.491176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:00.491272 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:00.491488 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:00.491490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:00.491676 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:00.491767 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:00.491772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.492474 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.492476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:00.493020 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:00.493027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.493438 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.493441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:00.493500 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:00.493950 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:00.493960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:00.495751 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":91,"threads_started":1}
W20260929 10:24:00.495993 30822 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:00.490961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:43289 (local address 127.21.33.190:46501)
0929 10:24:00.491098 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:00.491102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:00.491241 (+ 139us) server_negotiation.cc:407] Connection header received
0929 10:24:00.491348 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:00.491350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:00.491395 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:00.491453 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:00.491859 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.492376 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.493114 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:00.493348 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:00.493421 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:00.495881 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:43289: BlockingRecv error: recv got EOF from 127.21.33.131:43289 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 10:24:01 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:01 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:01.126986 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:01.120929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:51609)
0929 10:24:01.121283 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:01.121302 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:01.121406 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:01.121806 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:01.121809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:01.122084 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:01.122195 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:01.122201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.122949 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.122952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:01.123535 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:01.123543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.124048 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.124051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:01.124115 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:01.124688 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:01.124699 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:01.126869 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
W20260929 10:24:01.127182 30826 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:01.121283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:51609 (local address 127.21.33.190:46501)
0929 10:24:01.121533 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:01.121536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:01.121548 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:24:01.121583 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:01.121585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:01.121653 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:01.121746 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:01.122311 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.122843 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.123692 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.123938 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.124030 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:01.127007 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:51609: BlockingRecv error: recv got EOF from 127.21.33.129:51609 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
Sep 29 10:24:01 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:01 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:01.310650 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:01.305480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:35917)
0929 10:24:01.305776 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:01.305789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:01.305879 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:01.306116 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:01.306118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:01.306309 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:01.306398 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:01.306403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.307093 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.307095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:01.307655 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:01.307662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.308074 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.308078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:01.308177 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:01.308752 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:01.308764 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:01.310542 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
W20260929 10:24:01.310787 30826 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:01.305635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:35917 (local address 127.21.33.190:46501)
0929 10:24:01.305796 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:01.305799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:01.305839 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:24:01.305966 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:01.305968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:01.306012 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:01.306080 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:01.306495 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.307006 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.307756 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.307967 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.308043 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:01.310671 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:35917: BlockingRecv error: recv got EOF from 127.21.33.130:35917 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 10:24:01 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:01 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:01.502306 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:01.496809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:54043)
0929 10:24:01.497080 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:01.497094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:01.497270 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:01.497541 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:01.497544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:01.497758 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:01.497846 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:01.497852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.498618 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.498622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:01.499207 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:01.499216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.499714 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.499718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:01.499804 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:01.500297 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:01.500308 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:01.502183 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
W20260929 10:24:01.502445 30826 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:01.496863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:54043 (local address 127.21.33.190:46501)
0929 10:24:01.497019 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:01.497025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:01.497223 (+ 198us) server_negotiation.cc:407] Connection header received
0929 10:24:01.497387 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:01.497390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:01.497435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:01.497519 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:01.497968 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.498506 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.499331 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:01.499565 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:01.499692 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:01.502325 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:54043: BlockingRecv error: recv got EOF from 127.21.33.131:54043 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 10:24:02 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:02 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:02.133473 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:02.127806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:51063)
0929 10:24:02.128167 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:02.128182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:02.128322 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:02.128594 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:02.128597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:02.128808 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:02.128895 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:02.128901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.129720 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.129727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:02.130291 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:02.130299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.130750 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.130754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:02.130835 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:02.131352 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:02.131363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:02.133305 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":87,"threads_started":1}
W20260929 10:24:02.133548 30830 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:02.127871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:51063 (local address 127.21.33.190:46501)
0929 10:24:02.128148 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:02.128154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:02.128283 (+ 129us) server_negotiation.cc:407] Connection header received
0929 10:24:02.128422 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:02.128424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:02.128468 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:02.128569 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:02.129017 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.129615 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.130412 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.130638 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.130747 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:02.133422 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:51063: BlockingRecv error: recv got EOF from 127.21.33.129:51063 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
Sep 29 10:24:02 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:02 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:02.317787 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:02.311642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:58595)
0929 10:24:02.311963 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:02.311981 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:02.312124 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:02.312440 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:02.312443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:02.312678 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:02.312778 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:02.312784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.313707 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.313711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:02.314428 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:02.314440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.315083 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.315085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:02.315152 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:02.315735 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:02.315746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:02.317660 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":81,"threads_started":1}
W20260929 10:24:02.317996 30830 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:02.311808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:58595 (local address 127.21.33.190:46501)
0929 10:24:02.311967 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:02.311974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:02.312069 (+ 95us) server_negotiation.cc:407] Connection header received
0929 10:24:02.312238 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:02.312244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:02.312313 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:02.312405 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:02.312902 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.313576 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.314614 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.314958 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.315066 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:02.317869 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:58595: BlockingRecv error: recv got EOF from 127.21.33.130:58595 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 10:24:02 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:02 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:02.508919 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:02.503276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:40943)
0929 10:24:02.503553 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:02.503568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:02.503677 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:02.503927 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:02.503930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:02.504138 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:02.504235 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:02.504241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.504986 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.504989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:02.505739 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:02.505751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.506256 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.506260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:02.506329 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:02.506821 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:02.506832 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:02.508780 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
W20260929 10:24:02.509006 30830 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:02.503355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:40943 (local address 127.21.33.190:46501)
0929 10:24:02.503493 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:02.503498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:02.503627 (+ 129us) server_negotiation.cc:407] Connection header received
0929 10:24:02.503758 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:02.503760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:02.503809 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:02.503884 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:02.504338 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.504855 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.505892 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:02.506122 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:02.506218 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:02.508913 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:40943: BlockingRecv error: recv got EOF from 127.21.33.131:40943 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 10:24:03 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:03 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:03.139835 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:03.134410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:42261)
0929 10:24:03.134785 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:03.134799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:03.134924 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:03.135231 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:03.135234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:03.135477 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:03.135582 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:03.135589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.136320 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.136322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:03.136865 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:03.136872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.137354 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.137356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:03.137416 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:03.137901 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:03.137912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:03.139723 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
W20260929 10:24:03.140062 30833 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:03.134459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:42261 (local address 127.21.33.190:46501)
0929 10:24:03.134938 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:03.134942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:03.134956 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:24:03.135027 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:03.135030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:03.135082 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:03.135192 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:03.135688 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.136220 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.136991 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.137255 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.137344 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:03.139902 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:42261: BlockingRecv error: recv got EOF from 127.21.33.129:42261 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":70,"threads_started":1}
Sep 29 10:24:03 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:03 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:03.324412 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:03.318584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:54301)
0929 10:24:03.318913 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:03.318926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:03.319026 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:03.319283 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:03.319285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:03.319457 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:03.319550 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:03.319555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.320335 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.320337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:03.320982 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:03.320992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.321532 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.321535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:03.321631 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:03.322253 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:03.322276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:03.324298 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":85,"threads_started":1}
W20260929 10:24:03.324648 30833 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:03.318666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:54301 (local address 127.21.33.190:46501)
0929 10:24:03.318828 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:03.318834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:03.319002 (+ 168us) server_negotiation.cc:407] Connection header received
0929 10:24:03.319098 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:03.319101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:03.319150 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:03.319240 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:03.319669 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.320232 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.321103 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.321405 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.321516 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:03.324527 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:54301: BlockingRecv error: recv got EOF from 127.21.33.130:54301 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 10:24:03 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:03 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:03.515576 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:03.509855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:40379)
0929 10:24:03.510200 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:03.510212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:03.510363 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:03.510633 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:03.510636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:03.510841 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:03.510928 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:03.510934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.511695 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.511701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:03.512254 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:03.512269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.512730 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.512733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:03.512837 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:03.513428 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:03.513439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:03.515461 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":101,"threads_started":1}
W20260929 10:24:03.515833 30833 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:03.509938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:40379 (local address 127.21.33.190:46501)
0929 10:24:03.510107 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:03.510112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:03.510301 (+ 189us) server_negotiation.cc:407] Connection header received
0929 10:24:03.510469 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:03.510472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:03.510525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:03.510608 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:03.511044 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.511587 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.512380 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:03.512608 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:03.512727 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:03.515586 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:40379: BlockingRecv error: recv got EOF from 127.21.33.131:40379 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 10:24:04 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:04 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:04.146726 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:04.140942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:49697)
0929 10:24:04.141448 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:04.141466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:04.141581 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:04.141912 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:04.141915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:04.142150 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:04.142280 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:04.142288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.143054 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.143059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:04.143627 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:04.143635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.144098 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.144102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:04.144187 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:04.144742 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:04.144756 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:04.146559 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":89,"threads_started":1}
W20260929 10:24:04.146827 30838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:04.140990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:49697 (local address 127.21.33.190:46501)
0929 10:24:04.141291 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:04.141294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:04.141524 (+ 230us) server_negotiation.cc:407] Connection header received
0929 10:24:04.141702 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:04.141706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:04.141783 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:04.141885 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:04.142406 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.142943 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.143755 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.143989 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.144096 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:04.146684 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:49697: BlockingRecv error: recv got EOF from 127.21.33.129:49697 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":59,"threads_started":1}
Sep 29 10:24:04 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:04 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:04.331007 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:04.325188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:47779)
0929 10:24:04.325509 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:04.325522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:04.325655 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:04.325953 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:04.325956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:04.326185 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:04.326287 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:04.326293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.327187 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.327193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:04.327757 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:04.327766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.328225 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.328228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:04.328325 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:04.328912 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:04.328925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:04.330897 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
W20260929 10:24:04.331207 30838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:04.325272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:47779 (local address 127.21.33.190:46501)
0929 10:24:04.325433 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:04.325437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:04.325595 (+ 158us) server_negotiation.cc:407] Connection header received
0929 10:24:04.325781 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:04.325784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:04.325838 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:04.325927 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:04.326405 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.327077 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.327878 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.328111 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.328220 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:04.331035 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:47779: BlockingRecv error: recv got EOF from 127.21.33.130:47779 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 10:24:04 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:04 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:04.522027 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:04.516332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:60901)
0929 10:24:04.516579 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:04.516593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:04.516811 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:04.517079 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:04.517082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:04.517341 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:04.517464 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:04.517471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.518346 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.518348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:04.519040 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:04.519050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.519584 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.519586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:04.519687 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:04.520174 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:04.520185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:04.521914 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
W20260929 10:24:04.522212 30838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:04.516387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:60901 (local address 127.21.33.190:46501)
0929 10:24:04.516509 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:04.516513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:04.516754 (+ 241us) server_negotiation.cc:407] Connection header received
0929 10:24:04.516930 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:04.516933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:04.516993 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:04.517057 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:04.517596 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.518219 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.519217 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:04.519459 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:04.519548 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:04.522046 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:60901: BlockingRecv error: recv got EOF from 127.21.33.131:60901 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 10:24:05 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:05 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:05.158174 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:05.147520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:38219)
0929 10:24:05.147822 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:05.147835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:05.147926 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:05.148149 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:05.148151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:05.148323 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:05.148414 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:05.148420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.149218 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.149220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:05.149748 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:05.149755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.150191 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.150194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:05.150249 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:05.156051 (+ 5802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:05.156064 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:05.158058 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
W20260929 10:24:05.158380 30842 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:05.147572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:38219 (local address 127.21.33.190:46501)
0929 10:24:05.147852 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:05.147855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:05.147885 (+ 30us) server_negotiation.cc:407] Connection header received
0929 10:24:05.148004 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:05.148006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:05.148047 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:05.148122 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:05.148549 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.149091 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.149856 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.150081 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.150178 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:05.158218 (+ 8040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:38219: BlockingRecv error: recv got EOF from 127.21.33.129:38219 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
Sep 29 10:24:05 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:05 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:05.337355 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:05.331790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:58953)
0929 10:24:05.332084 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:05.332097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:05.332197 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:05.332429 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:05.332432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:05.332604 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:05.332722 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:05.332727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.333570 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.333573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:05.334120 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:05.334129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.334575 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.334578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:05.334645 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:05.335144 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:05.335155 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:05.337130 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
W20260929 10:24:05.337385 30842 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:05.331865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:58953 (local address 127.21.33.190:46501)
0929 10:24:05.332010 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:05.332013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:05.332169 (+ 156us) server_negotiation.cc:407] Connection header received
0929 10:24:05.332275 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:05.332278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:05.332321 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:05.332392 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:05.332862 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.333457 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.334244 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.334459 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.334541 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:05.337284 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:58953: BlockingRecv error: recv got EOF from 127.21.33.130:58953 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 10:24:05 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:05 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:05.528733 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:05.522846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:49565)
0929 10:24:05.523129 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:05.523142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:05.523275 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:05.523532 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:05.523535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:05.523743 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:05.523830 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:05.523836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.524602 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.524608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:05.525335 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:05.525344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.525875 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.525879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:05.525981 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:05.526661 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:05.526675 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:05.528609 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
W20260929 10:24:05.528927 30842 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:05.522895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:49565 (local address 127.21.33.190:46501)
0929 10:24:05.523053 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:05.523057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:05.523214 (+ 157us) server_negotiation.cc:407] Connection header received
0929 10:24:05.523372 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:05.523376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:05.523428 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:05.523514 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:05.523946 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.524489 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.525456 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:05.525734 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:05.525833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:05.528762 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:49565: BlockingRecv error: recv got EOF from 127.21.33.131:49565 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 10:24:06 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:06 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:06.164781 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:06.159095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:45011)
0929 10:24:06.159387 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:06.159412 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:06.159502 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:06.159785 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:06.159787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:06.159989 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:06.160088 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:06.160095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.160871 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.160874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:06.161547 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:06.161557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.162026 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.162029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:06.162095 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:06.162690 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:06.162702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:06.164593 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
W20260929 10:24:06.164863 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:06.159296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45011 (local address 127.21.33.190:46501)
0929 10:24:06.159523 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:06.159526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:06.159540 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:24:06.159594 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:06.159597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:06.159663 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:06.159743 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:06.160197 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.160752 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.161696 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.161924 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.162003 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:06.164729 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:45011: BlockingRecv error: recv got EOF from 127.21.33.129:45011 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
Sep 29 10:24:06 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:06 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:06.344148 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:06.338254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:34441)
0929 10:24:06.338543 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:06.338557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:06.338671 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:06.338945 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:06.338947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:06.339129 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:06.339224 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:06.339229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.340149 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.340152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:06.340885 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:06.340899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.341554 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.341558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:06.341648 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:06.342195 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:06.342206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:06.344019 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
W20260929 10:24:06.344281 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:06.338407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:34441 (local address 127.21.33.190:46501)
0929 10:24:06.338704 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:06.338708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:06.338723 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:24:06.338764 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:06.338766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:06.338806 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:06.338893 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:06.339357 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.339967 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.341014 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.341418 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.341522 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:06.344160 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:34441: BlockingRecv error: recv got EOF from 127.21.33.130:34441 (error 108)
Metrics: {"server-negotiator.queue_time_us":207}
Sep 29 10:24:06 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:06 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:06.535000 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:06.529536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:55495)
0929 10:24:06.529845 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:06.529862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:06.530000 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:06.530271 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:06.530274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:06.530475 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:06.530565 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:06.530571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.531324 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.531328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:06.531887 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:06.531894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.532360 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.532364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:06.532447 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:06.532923 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:06.532934 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:06.534832 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
W20260929 10:24:06.535058 30846 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:06.529707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:55495 (local address 127.21.33.190:46501)
0929 10:24:06.529845 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:06.529850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:06.529941 (+ 91us) server_negotiation.cc:407] Connection header received
0929 10:24:06.530097 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:06.530101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:06.530152 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:06.530246 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:06.530684 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.531216 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.532002 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:06.532248 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:06.532355 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:06.534949 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:55495: BlockingRecv error: recv got EOF from 127.21.33.131:55495 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 10:24:07 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:07 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:07.173543 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:07.165809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:57837)
0929 10:24:07.166168 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:07.166184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:07.166345 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:07.166728 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:07.166731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:07.167017 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:07.167165 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:07.167172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.168396 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.168399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:07.169411 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:07.169425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.170033 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.170037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:07.170131 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:07.170748 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:07.170766 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:07.173391 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
W20260929 10:24:07.173707 30850 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:07.165899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:57837 (local address 127.21.33.190:46501)
0929 10:24:07.166192 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:07.166199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:07.166298 (+ 99us) server_negotiation.cc:407] Connection header received
0929 10:24:07.166455 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:07.166460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:07.166525 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:07.166677 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:07.167320 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.168275 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.169537 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.169914 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.170022 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:07.173553 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:57837: BlockingRecv error: recv got EOF from 127.21.33.129:57837 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
Sep 29 10:24:07 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:07 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:07.351138 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:07.345016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:55427)
0929 10:24:07.345377 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:07.345400 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:07.345505 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:07.345817 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:07.345820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:07.346016 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:07.346136 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:07.346142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.346955 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.346958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:07.347552 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:07.347561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.348041 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.348045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:07.348116 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:07.348690 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:07.348702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:07.351012 (+ 2310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":212,"threads_started":1}
W20260929 10:24:07.351382 30850 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:07.345167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:55427 (local address 127.21.33.190:46501)
0929 10:24:07.345353 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:07.345357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:07.345477 (+ 120us) server_negotiation.cc:407] Connection header received
0929 10:24:07.345614 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:07.345617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:07.345663 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:07.345738 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:07.346291 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.346842 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.347677 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.347921 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.348002 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:07.351144 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:55427: BlockingRecv error: recv got EOF from 127.21.33.130:55427 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 10:24:07 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:07 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:07.541675 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:07.535881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:35781)
0929 10:24:07.536218 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:07.536236 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:07.536376 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:07.536663 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:07.536666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:07.536874 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:07.536962 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:07.536967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.537835 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.537841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:07.538411 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:07.538419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.539048 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.539051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:07.539121 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:07.539729 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:07.539744 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:07.541556 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
W20260929 10:24:07.541867 30850 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:07.535942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:35781 (local address 127.21.33.190:46501)
0929 10:24:07.536164 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:07.536168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:07.536316 (+ 148us) server_negotiation.cc:407] Connection header received
0929 10:24:07.536479 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:07.536482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:07.536547 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:07.536646 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:07.537085 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.537721 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.538535 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:07.538782 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:07.538894 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:07.541729 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:35781: BlockingRecv error: recv got EOF from 127.21.33.131:35781 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 29 10:24:08 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:08 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:08.179529 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:08.174357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:49201)
0929 10:24:08.174644 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:08.174667 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:08.174760 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:08.175007 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:08.175009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:08.175195 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:08.175284 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:08.175296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.176018 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.176020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:08.176555 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:08.176561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.177101 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.177104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:08.177169 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:08.177645 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:08.177656 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:08.179419 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
W20260929 10:24:08.179742 30854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:08.174498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:49201 (local address 127.21.33.190:46501)
0929 10:24:08.174741 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:08.174744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:08.174757 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:24:08.174845 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:08.174847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:08.174890 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:08.174966 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:08.175387 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.175928 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.176746 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.176974 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.177073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:08.179559 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:49201: BlockingRecv error: recv got EOF from 127.21.33.129:49201 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
Sep 29 10:24:08 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:08 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:08.357004 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:08.351895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:44427)
0929 10:24:08.352173 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:08.352186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:08.352287 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:08.352514 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:08.352516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:08.352692 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:08.352792 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:08.352797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.353572 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.353575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:08.354124 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:08.354132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.354544 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.354547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:08.354613 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:08.355111 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:08.355123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:08.356889 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
W20260929 10:24:08.357139 30854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:08.351949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:44427 (local address 127.21.33.190:46501)
0929 10:24:08.352102 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:08.352105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:08.352260 (+ 155us) server_negotiation.cc:407] Connection header received
0929 10:24:08.352365 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:08.352368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:08.352410 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:08.352478 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:08.352891 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.353477 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.354225 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.354439 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.354515 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:08.357022 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:44427: BlockingRecv error: recv got EOF from 127.21.33.130:44427 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 10:24:08 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:08 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:08.547744 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:08.542510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:56181)
0929 10:24:08.542807 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:08.542823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:08.542931 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:08.543216 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:08.543219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:08.543395 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:08.543502 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:08.543508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.544238 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.544241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:08.544804 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:08.544812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.545319 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.545326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:08.545389 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:08.545874 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:08.545885 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:08.547611 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":115,"threads_started":1}
W20260929 10:24:08.547916 30854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:08.542572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:56181 (local address 127.21.33.190:46501)
0929 10:24:08.542815 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:08.542818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:08.542904 (+ 86us) server_negotiation.cc:407] Connection header received
0929 10:24:08.543012 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:08.543014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:08.543060 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:08.543150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:08.543607 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.544143 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.544949 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:08.545191 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:08.545300 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:08.547782 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:56181: BlockingRecv error: recv got EOF from 127.21.33.131:56181 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 29 10:24:09 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:09 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:09.187112 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:09.180431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:49629)
0929 10:24:09.180719 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:09.180735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:09.180833 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:09.181119 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:09.181121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:09.181379 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:09.181480 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:09.181486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.182493 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.182499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:09.183329 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:09.183343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.183890 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.183893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:09.183980 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:09.184521 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:09.184532 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:09.186944 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
W20260929 10:24:09.187223 30858 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:09.180526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:49629 (local address 127.21.33.190:46501)
0929 10:24:09.180856 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:09.180859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:09.180876 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:24:09.180922 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:09.180924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:09.180973 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:09.181100 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:09.181620 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.182254 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.183464 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.183769 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.183886 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:09.187087 (+ 3201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:49629: BlockingRecv error: recv got EOF from 127.21.33.129:49629 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":136,"threads_started":1}
Sep 29 10:24:09 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:09 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:09.363808 30859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:09.357923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:41705)
0929 10:24:09.358532 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:09.358549 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:09.358670 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:09.359003 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:09.359006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:09.359252 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:09.359393 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:09.359401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.360186 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.360189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:09.360742 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:09.360750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.361186 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.361189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:09.361278 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:09.361819 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:09.361840 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:09.363647 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":133,"threads_started":1}
W20260929 10:24:09.363903 30858 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:09.358108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:41705 (local address 127.21.33.190:46501)
0929 10:24:09.358276 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:09.358280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:09.358637 (+ 357us) server_negotiation.cc:407] Connection header received
0929 10:24:09.358779 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:09.358783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:09.358841 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:09.358934 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:09.359522 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.360082 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.360851 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.361082 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.361172 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:09.363789 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:41705: BlockingRecv error: recv got EOF from 127.21.33.130:41705 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 10:24:09 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:09 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:09.554270 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:09.548639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:47755)
0929 10:24:09.548924 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:09.548939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:09.549046 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:09.549374 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:09.549376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:09.549600 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:09.549702 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:09.549708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.550483 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.550486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:09.551086 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:09.551095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.551532 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.551535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:09.551631 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:09.552141 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:09.552153 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:09.554148 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
W20260929 10:24:09.554443 30858 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:09.548735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:47755 (local address 127.21.33.190:46501)
0929 10:24:09.548885 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:09.548889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:09.548993 (+ 104us) server_negotiation.cc:407] Connection header received
0929 10:24:09.549116 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:09.549118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:09.549161 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:09.549238 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:09.549854 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.550378 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.551201 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:09.551416 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:09.551493 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:09.554313 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:47755: BlockingRecv error: recv got EOF from 127.21.33.131:47755 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 10:24:10 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:10 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:10.199072 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:10.187959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:45625)
0929 10:24:10.188254 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:10.188267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:10.188360 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:10.188692 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:10.188694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:10.188871 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:10.188963 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:10.188968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.189905 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.189908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:10.190556 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:10.190565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.191030 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.191033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:10.191095 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:10.196876 (+ 5781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:10.196891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:10.198955 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
W20260929 10:24:10.199268 30862 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:10.188134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45625 (local address 127.21.33.190:46501)
0929 10:24:10.188445 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:10.188448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:10.188461 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:24:10.188496 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:10.188499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:10.188538 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:10.188649 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:10.189091 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.189792 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.190704 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.190929 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.191013 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:10.199117 (+ 8104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:45625: BlockingRecv error: recv got EOF from 127.21.33.129:45625 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":94,"threads_started":1}
Sep 29 10:24:10 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:10 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:10.370030 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:10.364586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:59741)
0929 10:24:10.364884 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:10.364898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:10.364999 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:10.365273 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:10.365275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:10.365451 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:10.365556 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:10.365561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.366338 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.366341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:10.366952 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:10.366960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.367381 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.367383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:10.367449 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:10.367958 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:10.367969 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:10.369916 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
W20260929 10:24:10.370240 30862 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:10.364694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:59741 (local address 127.21.33.190:46501)
0929 10:24:10.364895 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:10.364898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:10.364971 (+ 73us) server_negotiation.cc:407] Connection header received
0929 10:24:10.365084 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:10.365087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:10.365137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:10.365230 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:10.365690 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.366216 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.367059 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.367278 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.367353 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:10.370097 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:59741: BlockingRecv error: recv got EOF from 127.21.33.130:59741 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 10:24:10 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:10 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.131@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:10.560580 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:10.555129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.131:50425)
0929 10:24:10.555392 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:10.555405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:10.555501 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:10.555761 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:10.555763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:10.555940 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:10.556034 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:10.556039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.556782 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.556784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:10.557440 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:10.557449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.557856 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.557858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:10.557918 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:10.558410 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:10.558421 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:10.560405 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
W20260929 10:24:10.560760 30862 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:10.555256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:50425 (local address 127.21.33.190:46501)
0929 10:24:10.555391 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:10.555394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:10.555473 (+ 79us) server_negotiation.cc:407] Connection header received
0929 10:24:10.555583 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:10.555586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:10.555629 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:10.555722 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:10.556132 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.556672 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.557540 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:10.557763 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:10.557840 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:10.560622 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.131:50425: BlockingRecv error: recv got EOF from 127.21.33.131:50425 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 10:24:11 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:11 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.129@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:11.205332 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:11.199955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.129:33973)
0929 10:24:11.200219 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:11.200233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:11.200316 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:11.200538 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:11.200540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:11.200742 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:11.200833 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:11.200839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.201579 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.201582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:11.202164 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:11.202172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.202611 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.202615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:11.202679 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:11.203281 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:11.203295 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:11.205143 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
W20260929 10:24:11.205502 30866 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:11.200087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:33973 (local address 127.21.33.190:46501)
0929 10:24:11.200330 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:11.200333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:11.200346 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:24:11.200384 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:11.200386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:11.200425 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:11.200506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:11.200936 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.201489 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.202271 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.202483 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.202592 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:11.205336 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.129:33973: BlockingRecv error: recv got EOF from 127.21.33.129:33973 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
Sep 29 10:24:11 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 10:24:11 dist-test-slave-2rbl krb5kdc[29808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.33.130@KRBTEST.COM for oryx/127.21.33.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 10:24:11.376323 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:11.371016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46501 (local address 127.21.33.130:46351)
0929 10:24:11.371317 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:11.371331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:11.371425 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:11.371697 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:11.371699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:11.371870 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:11.371974 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:11.371980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.372747 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.372750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:11.373397 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:11.373407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.373839 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.373841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:11.373900 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:11.374405 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:11.374416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:11.376211 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.33.190:46501: Server oryx/127.21.33.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
W20260929 10:24:11.376480 30866 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:11.371081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:46351 (local address 127.21.33.190:46501)
0929 10:24:11.371231 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:11.371236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:11.371404 (+ 168us) server_negotiation.cc:407] Connection header received
0929 10:24:11.371511 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:11.371514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:11.371557 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:11.371637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:11.372085 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.372634 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.373505 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.373741 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.373819 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:11.376356 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.33.130:46351: BlockingRecv error: recv got EOF from 127.21.33.130:46351 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:24:11.459553 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30391
I20260929 10:24:11.465044 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30543
I20260929 10:24:11.470275 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30675
I20260929 10:24:11.475596 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30321
2026-09-29T10:24:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18015 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T10:24:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:24:11Z Disabled control of system clock
I20260929 10:24:11.505409 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:37109
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:37109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 10:24:11.615273 30871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:11.615543 30871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:11.615584 30871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:11.619518 30871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:24:11.619581 30871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:11.619597 30871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:11.619616 30871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:24:11.619676 30871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:24:11.624436 30871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:37109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:11.625476 30871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:11.626807 30871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:11.635710 30877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:11.635787 30876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:11.635902 30871 server_base.cc:1034] running on GCE node
W20260929 10:24:11.635731 30879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:11.636590 30871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:11.637805 30871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:11.639004 30871 hybrid_clock.cc:648] HybridClock initialized: now 1790677451638998 us; error 44 us; skew 500 ppm
I20260929 10:24:11.640908 30871 webserver.cc:460] Webserver started at http://127.21.33.190:33261/ using document root <none> and password file <none>
I20260929 10:24:11.641580 30871 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:11.641670 30871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:11.641902 30871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:11.643925 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "5d081df308ab4947adedeccd82e4a5e4"
format_stamp: "Formatted at 2026-09-29 10:24:11 on dist-test-slave-2rbl"
server_key: "ad79dcbec04420fd3a10e7dab5d89834"
server_key_iv: "1a3401e035d3b406ac891e114575cb1f"
server_key_version: "encryptionkey@0"
I20260929 10:24:11.644512 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "5d081df308ab4947adedeccd82e4a5e4"
format_stamp: "Formatted at 2026-09-29 10:24:11 on dist-test-slave-2rbl"
server_key: "ad79dcbec04420fd3a10e7dab5d89834"
server_key_iv: "1a3401e035d3b406ac891e114575cb1f"
server_key_version: "encryptionkey@0"
I20260929 10:24:11.648272 30871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:24:11.650880 30885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:11.652127 30871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:24:11.652287 30871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "5d081df308ab4947adedeccd82e4a5e4"
format_stamp: "Formatted at 2026-09-29 10:24:11 on dist-test-slave-2rbl"
server_key: "ad79dcbec04420fd3a10e7dab5d89834"
server_key_iv: "1a3401e035d3b406ac891e114575cb1f"
server_key_version: "encryptionkey@0"
I20260929 10:24:11.652406 30871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:11.675787 30871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:11.676508 30871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:11.676716 30871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:11.683367 30871 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:37109
I20260929 10:24:11.683378 30937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:37109 every 8 connection(s)
I20260929 10:24:11.684274 30871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:24:11.687405 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:11.691028 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30871
I20260929 10:24:11.691152 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:24:11.691494 21638 external_mini_cluster.cc:1428] Setting key 8753f694ea6e0ad7103acdf09ff2b21e
I20260929 10:24:11.693107 30938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4: Bootstrap starting.
I20260929 10:24:11.695322 30938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:11.696059 30938 log.cc:824] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:11.698045 30938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4: No bootstrap required, opened a new log
I20260929 10:24:11.701089 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:11.693021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.21.33.190:37109)
0929 10:24:11.693735 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:11.693752 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:11.693788 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:24:11.694428 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:11.694445 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:11.694846 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:11.695187 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:11.695628 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.696788 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.697799 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.698340 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.698511 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:11.699394 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:11.699400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:11.699422 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:11.699799 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:11.699854 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:11.699873 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:11.699982 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:11.700110 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 10:24:11.700374 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":152,"threads_started":1}
I20260929 10:24:11.701663 30938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d081df308ab4947adedeccd82e4a5e4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37109 } }
I20260929 10:24:11.701910 30938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:11.701990 30938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d081df308ab4947adedeccd82e4a5e4, State: Initialized, Role: FOLLOWER
I20260929 10:24:11.702418 30938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d081df308ab4947adedeccd82e4a5e4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37109 } }
I20260929 10:24:11.702572 30938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:24:11.702670 30938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:24:11.702790 30938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:24:11.703861 30938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d081df308ab4947adedeccd82e4a5e4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37109 } }
I20260929 10:24:11.704200 30938 leader_election.cc:302] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d081df308ab4947adedeccd82e4a5e4; no voters:
I20260929 10:24:11.704495 30938 leader_election.cc:288] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:24:11.704643 30943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:24:11.704865 30943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d081df308ab4947adedeccd82e4a5e4, State: Running, Role: LEADER
I20260929 10:24:11.705276 30943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d081df308ab4947adedeccd82e4a5e4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37109 } }
I20260929 10:24:11.705796 30938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:24:11.706873 30944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d081df308ab4947adedeccd82e4a5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d081df308ab4947adedeccd82e4a5e4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37109 } } }
I20260929 10:24:11.706996 30944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:24:11.707392 30952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:24:11.707376 30945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d081df308ab4947adedeccd82e4a5e4. Latest consensus state: current_term: 1 leader_uuid: "5d081df308ab4947adedeccd82e4a5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d081df308ab4947adedeccd82e4a5e4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37109 } } }
I20260929 10:24:11.707522 30945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4 [sys.catalog]: This master's current role is: LEADER
I20260929 10:24:11.710405 30952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:24:11.715703 30952 catalog_manager.cc:1347] Generated new cluster ID: 066a8e295f0241e7b6072876cd0fbfa4
I20260929 10:24:11.715770 30952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:24:11.727596 30952 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:24:11.728561 30952 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:24:11.738387 30952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d081df308ab4947adedeccd82e4a5e4: Generated new TSK 0
I20260929 10:24:11.739379 30952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:24:11.748085 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:37109
--builtin_ntp_servers=127.21.33.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 10:24:11.858734 30962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:11.859014 30962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:11.859081 30962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:11.863061 30962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:11.863140 30962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:11.863252 30962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:24:11.867890 30962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:11.869277 30962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:11.870623 30962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:11.880157 30970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:11.880169 30967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:11.880142 30968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:11.880426 30962 server_base.cc:1034] running on GCE node
I20260929 10:24:11.880874 30962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:11.881635 30962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:11.882833 30962 hybrid_clock.cc:648] HybridClock initialized: now 1790677451882793 us; error 51 us; skew 500 ppm
I20260929 10:24:11.884706 30962 webserver.cc:460] Webserver started at http://127.21.33.129:44315/ using document root <none> and password file <none>
I20260929 10:24:11.885414 30962 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:11.885510 30962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:11.885751 30962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:11.887844 30962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "fc68cc7f20a941f4b5ddc9c12f277262"
format_stamp: "Formatted at 2026-09-29 10:24:11 on dist-test-slave-2rbl"
server_key: "3e62eb0c48b3b8eff3ab3ac2ad00521b"
server_key_iv: "88aca5f821d03d65ef77f90f391b1b9e"
server_key_version: "encryptionkey@0"
I20260929 10:24:11.888432 30962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "fc68cc7f20a941f4b5ddc9c12f277262"
format_stamp: "Formatted at 2026-09-29 10:24:11 on dist-test-slave-2rbl"
server_key: "3e62eb0c48b3b8eff3ab3ac2ad00521b"
server_key_iv: "88aca5f821d03d65ef77f90f391b1b9e"
server_key_version: "encryptionkey@0"
I20260929 10:24:11.892294 30962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:24:11.894928 30976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:11.896188 30962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:24:11.896322 30962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "fc68cc7f20a941f4b5ddc9c12f277262"
format_stamp: "Formatted at 2026-09-29 10:24:11 on dist-test-slave-2rbl"
server_key: "3e62eb0c48b3b8eff3ab3ac2ad00521b"
server_key_iv: "88aca5f821d03d65ef77f90f391b1b9e"
server_key_version: "encryptionkey@0"
I20260929 10:24:11.896440 30962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:11.929031 30962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:11.929833 30962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:11.930039 30962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:11.930660 30962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:11.931622 30962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:11.931694 30962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:11.931777 30962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:11.931866 30962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:11.940577 30962 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:38003
I20260929 10:24:11.940629 31089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:38003 every 8 connection(s)
I20260929 10:24:11.941584 30962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:24:11.944734 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 30962
I20260929 10:24:11.944859 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:24:11.945220 21638 external_mini_cluster.cc:1428] Setting key 1448c126629992c5d98110e8872a7831
I20260929 10:24:11.947328 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:37109
--builtin_ntp_servers=127.21.33.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 10:24:11.952692 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:11.943604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:48025 (local address 127.21.33.190:37109)
0929 10:24:11.943772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:11.943776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:11.945036 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 10:24:11.946843 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:11.946845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:11.946886 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:11.946957 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:11.948240 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.948935 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.950523 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.950773 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.950889 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:11.952165 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:11.952168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:11.952170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:11.952182 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:11.952203 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:11.952211 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:11.952291 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:11.952466 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 10:24:11.952531 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 10:24:11.953641 31092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:11.943972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:37109 (local address 127.21.33.129:48025)
0929 10:24:11.944849 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:11.944909 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:11.946577 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:11.947166 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:11.947178 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:11.947297 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:11.948054 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:11.948076 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.949090 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.949096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:11.950382 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:11.950391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:11.950930 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:11.950934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:11.951180 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:11.951852 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:24:11.951854 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:11.951894 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:24:11.951959 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:11.952306 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:11.952315 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:11.952321 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:24:11.952465 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 10:24:11.952758 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":102,"threads_started":1}
I20260929 10:24:11.954932 31090 heartbeater.cc:347] Connected to a master server at 127.21.33.190:37109
I20260929 10:24:11.955189 31090 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:11.955792 31090 heartbeater.cc:511] Master 127.21.33.190:37109 requested a full tablet report, sending...
I20260929 10:24:11.957458 30902 ts_manager.cc:194] Registered new tserver with Master: fc68cc7f20a941f4b5ddc9c12f277262 (127.21.33.129:38003)
I20260929 10:24:11.958604 30902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.129:48025
W20260929 10:24:12.057194 31093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:12.057483 31093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:12.057549 31093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:12.061532 31093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:12.061606 31093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:12.061691 31093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:24:12.066411 31093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:12.067490 31093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:12.068882 31093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:12.079102 31101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.079165 31099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.079142 31098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:12.079133 31093 server_base.cc:1034] running on GCE node
I20260929 10:24:12.079726 31093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:12.080400 31093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:12.081588 31093 hybrid_clock.cc:648] HybridClock initialized: now 1790677452081571 us; error 34 us; skew 500 ppm
I20260929 10:24:12.083442 31093 webserver.cc:460] Webserver started at http://127.21.33.130:34813/ using document root <none> and password file <none>
I20260929 10:24:12.084041 31093 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:12.084131 31093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:12.084352 31093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:12.086483 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "58d7e6087a0c488898540c5e486d75e1"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "95207b962a575c448a825d739ec23afe"
server_key_iv: "cf2f5b04ddfed4611a533cfe57916bc8"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.087057 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "58d7e6087a0c488898540c5e486d75e1"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "95207b962a575c448a825d739ec23afe"
server_key_iv: "cf2f5b04ddfed4611a533cfe57916bc8"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.090786 31093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:24:12.093349 31107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.094540 31093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:24:12.094694 31093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "58d7e6087a0c488898540c5e486d75e1"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "95207b962a575c448a825d739ec23afe"
server_key_iv: "cf2f5b04ddfed4611a533cfe57916bc8"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.094813 31093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:12.106739 31093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:12.107425 31093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:12.107637 31093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:12.108278 31093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:12.109242 31093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:12.109308 31093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.109381 31093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:12.109414 31093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.118036 31093 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:43283
I20260929 10:24:12.118062 31220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:43283 every 8 connection(s)
I20260929 10:24:12.118968 31093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:24:12.123960 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31093
I20260929 10:24:12.124089 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:24:12.124393 21638 external_mini_cluster.cc:1428] Setting key bf0a51bc007d766ea0a87759b4e810d4
I20260929 10:24:12.126528 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:37109
--builtin_ntp_servers=127.21.33.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 10:24:12.129179 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.120912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:54949 (local address 127.21.33.190:37109)
0929 10:24:12.121081 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:12.121085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:12.122178 (+ 1093us) server_negotiation.cc:407] Connection header received
0929 10:24:12.123743 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:12.123749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:12.123812 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:12.123904 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:12.125118 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.125756 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.127090 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.127314 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.127430 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:12.128721 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:12.128724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:12.128726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:12.128738 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:12.128757 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:12.128765 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:12.128825 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:12.129005 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 10:24:12.129056 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:24:12.130381 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.121308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:37109 (local address 127.21.33.130:54949)
0929 10:24:12.122019 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:12.122068 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:12.123482 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:12.124124 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:12.124138 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:12.124258 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:12.124964 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:12.124988 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.125894 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.125897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:12.126950 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:12.126959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.127467 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.127471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:12.127739 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:12.128395 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:24:12.128398 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:12.128457 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:24:12.128550 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:12.128850 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:12.128860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:12.128869 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 10:24:12.129002 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 10:24:12.129320 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":101,"threads_started":1}
I20260929 10:24:12.131582 31221 heartbeater.cc:347] Connected to a master server at 127.21.33.190:37109
I20260929 10:24:12.131820 31221 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:12.132421 31221 heartbeater.cc:511] Master 127.21.33.190:37109 requested a full tablet report, sending...
I20260929 10:24:12.133523 30902 ts_manager.cc:194] Registered new tserver with Master: 58d7e6087a0c488898540c5e486d75e1 (127.21.33.130:43283)
I20260929 10:24:12.134101 30902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.130:54949
W20260929 10:24:12.238417 31224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:12.238706 31224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:12.238804 31224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:12.242863 31224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:12.242947 31224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:12.243033 31224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:24:12.247885 31224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:37109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:12.249039 31224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:12.250520 31224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:24:12.260824 31224 server_base.cc:1034] running on GCE node
W20260929 10:24:12.260695 31230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.260747 31229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.260726 31232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:12.261375 31224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:12.261992 31224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:12.263147 31224 hybrid_clock.cc:648] HybridClock initialized: now 1790677452263133 us; error 26 us; skew 500 ppm
I20260929 10:24:12.264959 31224 webserver.cc:460] Webserver started at http://127.21.33.131:35895/ using document root <none> and password file <none>
I20260929 10:24:12.265605 31224 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:12.265671 31224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:12.265924 31224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:12.267935 31224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "0de1d74ab6b84b0fa090b3cc07c84a24"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "845b256de1511fc629e8686a34eb4d98"
server_key_iv: "e8a7c5c867cf94ba73f46d50d99e1e85"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.268503 31224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "0de1d74ab6b84b0fa090b3cc07c84a24"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "845b256de1511fc629e8686a34eb4d98"
server_key_iv: "e8a7c5c867cf94ba73f46d50d99e1e85"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.272255 31224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:24:12.275069 31238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.276427 31224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:24:12.276597 31224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "0de1d74ab6b84b0fa090b3cc07c84a24"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "845b256de1511fc629e8686a34eb4d98"
server_key_iv: "e8a7c5c867cf94ba73f46d50d99e1e85"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.276719 31224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:12.296137 31224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:12.296861 31224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:12.297012 31224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:12.297720 31224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:12.298748 31224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:12.298830 31224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.298888 31224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:12.298920 31224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.308197 31224 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:34945
I20260929 10:24:12.308286 31351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:34945 every 8 connection(s)
I20260929 10:24:12.309129 31224 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:24:12.313578 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31224
I20260929 10:24:12.313707 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:24:12.313975 21638 external_mini_cluster.cc:1428] Setting key ae710f47cb7b35ec03c242401ec167b2
I20260929 10:24:12.318499 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.310914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:38551 (local address 127.21.33.190:37109)
0929 10:24:12.311047 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:12.311050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:12.312064 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 10:24:12.313467 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:12.313471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:12.313532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:12.313641 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:12.314635 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.315341 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.316708 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.316918 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.316998 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:12.318017 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:12.318021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:12.318023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:12.318034 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:12.318067 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:12.318075 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:12.318136 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:12.318348 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 10:24:12.318395 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:24:12.319345 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.311274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:37109 (local address 127.21.33.131:38551)
0929 10:24:12.311914 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:12.311953 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:12.313243 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:12.313831 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:12.313840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:12.313932 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:12.314478 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:12.314495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.315459 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.315462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:12.316572 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:12.316581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:12.317043 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:12.317045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:12.317180 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:12.317774 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:24:12.317775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:12.317815 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:24:12.317874 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:12.318187 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:12.318195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:12.318200 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:24:12.318328 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 10:24:12.318537 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260929 10:24:12.320393 31352 heartbeater.cc:347] Connected to a master server at 127.21.33.190:37109
I20260929 10:24:12.320583 31352 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:12.321009 31352 heartbeater.cc:511] Master 127.21.33.190:37109 requested a full tablet report, sending...
I20260929 10:24:12.322052 30902 ts_manager.cc:194] Registered new tserver with Master: 0de1d74ab6b84b0fa090b3cc07c84a24 (127.21.33.131:34945)
I20260929 10:24:12.322691 30902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.131:38551
I20260929 10:24:12.327816 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:24:12.330480 30941 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:12.329626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.21.33.190:37109)
0929 10:24:12.329756 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:12.329759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:12.329944 (+ 185us) server_negotiation.cc:407] Connection header received
0929 10:24:12.330035 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:12.330038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:12.330073 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:12.330138 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:12.330343 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38936: BlockingRecv error: recv got EOF from 127.0.0.1:38936 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:24:12.330937 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30962
I20260929 10:24:12.337047 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31093
I20260929 10:24:12.342381 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31224
I20260929 10:24:12.347502 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 30871
2026-09-29T10:24:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (871 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T10:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:24:12Z Disabled control of system clock
I20260929 10:24:12.376848 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:36871
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 10:24:12.489514 31363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:12.489817 31363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:12.489856 31363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:12.494038 31363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:24:12.494102 31363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:12.494119 31363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:12.494136 31363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:24:12.494151 31363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:24:12.498946 31363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:36871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:12.500056 31363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:12.501441 31363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:12.510560 31369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:12.510725 31363 server_base.cc:1034] running on GCE node
W20260929 10:24:12.510620 31371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.510596 31368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:12.511411 31363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:12.512598 31363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:12.513790 31363 hybrid_clock.cc:648] HybridClock initialized: now 1790677452513753 us; error 51 us; skew 500 ppm
I20260929 10:24:12.515789 31363 webserver.cc:460] Webserver started at http://127.21.33.190:33897/ using document root <none> and password file <none>
I20260929 10:24:12.516381 31363 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:12.516474 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:12.516733 31363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:12.518851 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "f6186afd2c95455ea948f50766cc067a"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "9358079247d7bcb9a3c5d4968a748198"
server_key_iv: "4e6f0062070a9ea4478f8cf1542c3a9e"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.519421 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "f6186afd2c95455ea948f50766cc067a"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "9358079247d7bcb9a3c5d4968a748198"
server_key_iv: "4e6f0062070a9ea4478f8cf1542c3a9e"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.523221 31363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:24:12.525695 31377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.526917 31363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:24:12.527060 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "f6186afd2c95455ea948f50766cc067a"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "9358079247d7bcb9a3c5d4968a748198"
server_key_iv: "4e6f0062070a9ea4478f8cf1542c3a9e"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.527179 31363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:12.531695 31363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:12.532280 31363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:12.532464 31363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:12.539433 31363 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:36871
I20260929 10:24:12.539450 31429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:36871 every 8 connection(s)
I20260929 10:24:12.540382 31363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:24:12.542773 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31363
I20260929 10:24:12.542915 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:24:12.543205 21638 external_mini_cluster.cc:1428] Setting key b9722db86dfd969389effebca05eabb2
I20260929 10:24:12.543632 31430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:12.548735 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.544538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.21.33.190:36871)
0929 10:24:12.545180 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:12.545239 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:12.545287 (+ 48us) server_negotiation.cc:407] Connection header received
0929 10:24:12.545934 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:12.545980 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:12.546332 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:12.546662 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:12.547514 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:12.547518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:12.547530 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:12.547540 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:12.547594 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:12.547610 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:12.547762 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:12.547829 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:24:12.548067 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
I20260929 10:24:12.550408 31430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a: Bootstrap starting.
I20260929 10:24:12.552634 31430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:12.553428 31430 log.cc:824] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:12.555333 31430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a: No bootstrap required, opened a new log
I20260929 10:24:12.558866 31430 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6186afd2c95455ea948f50766cc067a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36871 } }
I20260929 10:24:12.559113 31430 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:12.559195 31430 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6186afd2c95455ea948f50766cc067a, State: Initialized, Role: FOLLOWER
I20260929 10:24:12.559669 31430 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6186afd2c95455ea948f50766cc067a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36871 } }
I20260929 10:24:12.559823 31430 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:24:12.559911 31430 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:24:12.560010 31430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:24:12.560892 31430 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6186afd2c95455ea948f50766cc067a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36871 } }
I20260929 10:24:12.561321 31430 leader_election.cc:302] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6186afd2c95455ea948f50766cc067a; no voters:
I20260929 10:24:12.561626 31430 leader_election.cc:288] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:24:12.561772 31435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:24:12.562009 31435 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [term 1 LEADER]: Becoming Leader. State: Replica: f6186afd2c95455ea948f50766cc067a, State: Running, Role: LEADER
I20260929 10:24:12.562304 31435 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6186afd2c95455ea948f50766cc067a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36871 } }
I20260929 10:24:12.563019 31430 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:24:12.564040 31436 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6186afd2c95455ea948f50766cc067a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6186afd2c95455ea948f50766cc067a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36871 } } }
I20260929 10:24:12.564177 31436 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [sys.catalog]: This master's current role is: LEADER
I20260929 10:24:12.564538 31444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:24:12.564519 31437 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6186afd2c95455ea948f50766cc067a. Latest consensus state: current_term: 1 leader_uuid: "f6186afd2c95455ea948f50766cc067a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6186afd2c95455ea948f50766cc067a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36871 } } }
I20260929 10:24:12.565295 31437 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a [sys.catalog]: This master's current role is: LEADER
I20260929 10:24:12.567595 31444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:24:12.573045 31444 catalog_manager.cc:1347] Generated new cluster ID: 25a876072f4347ef96b4390c368a7910
I20260929 10:24:12.573141 31444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:24:12.587617 31444 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:24:12.587975 31444 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:24:12.593246 31444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6186afd2c95455ea948f50766cc067a: Generated new TSK 0
I20260929 10:24:12.594157 31444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:24:12.607120 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36871
--builtin_ntp_servers=127.21.33.148:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 10:24:12.718300 31454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:12.718551 31454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:12.718592 31454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:12.722610 31454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:12.722672 31454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:12.722730 31454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:24:12.727438 31454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:12.728477 31454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:12.729835 31454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:12.739917 31460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:12.739993 31454 server_base.cc:1034] running on GCE node
W20260929 10:24:12.740206 31462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.740550 31459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:12.741086 31454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:12.741797 31454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:12.742981 31454 hybrid_clock.cc:648] HybridClock initialized: now 1790677452742961 us; error 37 us; skew 500 ppm
I20260929 10:24:12.744792 31454 webserver.cc:460] Webserver started at http://127.21.33.129:36631/ using document root <none> and password file <none>
I20260929 10:24:12.745426 31454 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:12.745489 31454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:12.745718 31454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:12.747763 31454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "a874fc7641134a6596c48a7a364354fa"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "b2f32cdd262c709be0f698ac3216a331"
server_key_iv: "d23e017267cfd2be850e7955daa23f0c"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.748308 31454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "a874fc7641134a6596c48a7a364354fa"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "b2f32cdd262c709be0f698ac3216a331"
server_key_iv: "d23e017267cfd2be850e7955daa23f0c"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.751864 31454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 10:24:12.754395 31468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.755555 31454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 10:24:12.755693 31454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "a874fc7641134a6596c48a7a364354fa"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "b2f32cdd262c709be0f698ac3216a331"
server_key_iv: "d23e017267cfd2be850e7955daa23f0c"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.755805 31454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:12.760320 31454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:12.760913 31454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:12.761094 31454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:12.761658 31454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:12.762527 31454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:12.762596 31454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.762686 31454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:12.762737 31454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.771441 31454 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:36751
I20260929 10:24:12.771468 31581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:36751 every 8 connection(s)
I20260929 10:24:12.772290 31454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:24:12.773309 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31454
I20260929 10:24:12.773438 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:24:12.773726 21638 external_mini_cluster.cc:1428] Setting key 98d906f70c065ab1cadcb286183c891b
I20260929 10:24:12.775835 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36871
--builtin_ntp_servers=127.21.33.148:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 10:24:12.778826 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.774327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:59591 (local address 127.21.33.190:36871)
0929 10:24:12.774488 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:12.774493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:12.775641 (+ 1148us) server_negotiation.cc:407] Connection header received
0929 10:24:12.777009 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:12.777012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:12.777053 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:12.777139 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:12.778255 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:12.778261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:12.778267 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:12.778301 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:12.778326 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:12.778339 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:12.778427 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:12.778594 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 10:24:12.778642 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:24:12.779567 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.774767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36871 (local address 127.21.33.129:59591)
0929 10:24:12.775498 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:12.775538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:12.776805 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:12.777310 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:12.777316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:12.777404 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:12.778003 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:24:12.778005 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:12.778043 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:24:12.778108 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:12.778446 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:12.778454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:12.778459 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:24:12.778605 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 10:24:12.778795 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":149,"threads_started":1}
I20260929 10:24:12.780790 31582 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36871
I20260929 10:24:12.781028 31582 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:12.781483 31582 heartbeater.cc:511] Master 127.21.33.190:36871 requested a full tablet report, sending...
I20260929 10:24:12.782963 31394 ts_manager.cc:194] Registered new tserver with Master: a874fc7641134a6596c48a7a364354fa (127.21.33.129:36751)
W20260929 10:24:12.897922 31585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:12.898345 31585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:12.898420 31585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:12.906497 31585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:12.906581 31585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:12.906704 31585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:24:12.915769 31585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:12.916811 31585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:12.918470 31585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:12.928151 31593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.928164 31591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:12.928179 31590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:12.928336 31585 server_base.cc:1034] running on GCE node
I20260929 10:24:12.928819 31585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:12.929509 31585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:12.930675 31585 hybrid_clock.cc:648] HybridClock initialized: now 1790677452930658 us; error 27 us; skew 500 ppm
I20260929 10:24:12.932628 31585 webserver.cc:460] Webserver started at http://127.21.33.130:33083/ using document root <none> and password file <none>
I20260929 10:24:12.933176 31585 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:12.933314 31585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:12.933501 31585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:12.935734 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "e32bab3c49614b8ba3bfa6a4a0cfead3"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "68432c1a3cf9a772d7b96434f40bf6ac"
server_key_iv: "15a684a5625639f4b2d4ec41482c337b"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.936348 31585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "e32bab3c49614b8ba3bfa6a4a0cfead3"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "68432c1a3cf9a772d7b96434f40bf6ac"
server_key_iv: "15a684a5625639f4b2d4ec41482c337b"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.940126 31585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:24:12.942854 31599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.944168 31585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:24:12.944329 31585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "e32bab3c49614b8ba3bfa6a4a0cfead3"
format_stamp: "Formatted at 2026-09-29 10:24:12 on dist-test-slave-2rbl"
server_key: "68432c1a3cf9a772d7b96434f40bf6ac"
server_key_iv: "15a684a5625639f4b2d4ec41482c337b"
server_key_version: "encryptionkey@0"
I20260929 10:24:12.944453 31585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:12.949170 31585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:12.949824 31585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:12.950004 31585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:12.950539 31585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:12.951426 31585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:12.951494 31585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.951571 31585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:12.951624 31585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:12.960583 31585 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:34889
I20260929 10:24:12.960599 31712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:34889 every 8 connection(s)
I20260929 10:24:12.961536 31585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:24:12.966735 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31585
I20260929 10:24:12.966866 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:24:12.967204 21638 external_mini_cluster.cc:1428] Setting key 4269063016d38d58fd934e1ede21dc86
I20260929 10:24:12.967705 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.963332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:52703 (local address 127.21.33.190:36871)
0929 10:24:12.963495 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:12.963499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:12.964615 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 10:24:12.965966 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:12.965968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:12.966004 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:12.966085 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:12.967215 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:12.967222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:12.967225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:12.967240 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:12.967264 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:12.967274 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:12.967375 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:12.967517 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 10:24:12.967594 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:24:12.968770 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:12.963763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36871 (local address 127.21.33.130:52703)
0929 10:24:12.964466 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:12.964504 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:12.965782 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:12.966231 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:12.966237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:12.966319 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:12.966930 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:24:12.966932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:12.966994 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:24:12.967068 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:12.967365 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:12.967373 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:12.967378 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:24:12.967521 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 10:24:12.967792 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260929 10:24:12.969493 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36871
--builtin_ntp_servers=127.21.33.148:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 10:24:12.969986 31713 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36871
I20260929 10:24:12.970227 31713 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:12.970695 31713 heartbeater.cc:511] Master 127.21.33.190:36871 requested a full tablet report, sending...
I20260929 10:24:12.971822 31394 ts_manager.cc:194] Registered new tserver with Master: e32bab3c49614b8ba3bfa6a4a0cfead3 (127.21.33.130:34889)
W20260929 10:24:13.080865 31716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:13.081156 31716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:13.081255 31716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:13.085400 31716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:13.085474 31716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:13.085565 31716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:24:13.090302 31716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:13.091440 31716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:13.092830 31716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:13.102466 31721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:13.102597 31724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:13.102443 31722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:13.102715 31716 server_base.cc:1034] running on GCE node
I20260929 10:24:13.103097 31716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:13.103725 31716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:13.104885 31716 hybrid_clock.cc:648] HybridClock initialized: now 1790677453104872 us; error 26 us; skew 500 ppm
I20260929 10:24:13.106727 31716 webserver.cc:460] Webserver started at http://127.21.33.131:39747/ using document root <none> and password file <none>
I20260929 10:24:13.107390 31716 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:13.107453 31716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:13.107673 31716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:13.109746 31716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "b5d456a44d1a40debe853d10cbcf3762"
format_stamp: "Formatted at 2026-09-29 10:24:13 on dist-test-slave-2rbl"
server_key: "68f0a7248521c13146bb0052f823d98e"
server_key_iv: "3fb975e4f87c8902979fe804ed73a68b"
server_key_version: "encryptionkey@0"
I20260929 10:24:13.110289 31716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "b5d456a44d1a40debe853d10cbcf3762"
format_stamp: "Formatted at 2026-09-29 10:24:13 on dist-test-slave-2rbl"
server_key: "68f0a7248521c13146bb0052f823d98e"
server_key_iv: "3fb975e4f87c8902979fe804ed73a68b"
server_key_version: "encryptionkey@0"
I20260929 10:24:13.113895 31716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:24:13.116297 31730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:13.117619 31716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:24:13.117750 31716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "b5d456a44d1a40debe853d10cbcf3762"
format_stamp: "Formatted at 2026-09-29 10:24:13 on dist-test-slave-2rbl"
server_key: "68f0a7248521c13146bb0052f823d98e"
server_key_iv: "3fb975e4f87c8902979fe804ed73a68b"
server_key_version: "encryptionkey@0"
I20260929 10:24:13.117853 31716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:13.122243 31716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:13.122826 31716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:13.123001 31716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:13.123523 31716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:13.124483 31716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:13.124543 31716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:13.124593 31716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:13.124617 31716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:13.133018 31716 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:37473
I20260929 10:24:13.133087 31843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:37473 every 8 connection(s)
I20260929 10:24:13.133953 31716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:24:13.136073 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31716
I20260929 10:24:13.136209 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:24:13.136492 21638 external_mini_cluster.cc:1428] Setting key 42da8d0eaf0beb1b6c912a78d209f3a4
I20260929 10:24:13.140370 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:13.135960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:36519 (local address 127.21.33.190:36871)
0929 10:24:13.136135 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:13.136140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:13.137420 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 10:24:13.138835 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:13.138838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:13.138871 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:13.138936 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:13.139990 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:13.139993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:13.139995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:24:13.140006 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:13.140024 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:24:13.140032 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:13.140117 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:13.140232 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 10:24:13.140273 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:24:13.141319 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:13.136511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36871 (local address 127.21.33.131:36519)
0929 10:24:13.137283 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:13.137334 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:13.138659 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:13.139067 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:13.139073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:13.139160 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:13.139732 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:24:13.139733 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:13.139773 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:24:13.139839 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:13.140125 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:13.140131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:13.140136 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 10:24:13.140261 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 10:24:13.140446 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260929 10:24:13.142350 31844 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36871
I20260929 10:24:13.142603 31844 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:13.143023 31844 heartbeater.cc:511] Master 127.21.33.190:36871 requested a full tablet report, sending...
I20260929 10:24:13.144003 31394 ts_manager.cc:194] Registered new tserver with Master: b5d456a44d1a40debe853d10cbcf3762 (127.21.33.131:37473)
I20260929 10:24:13.150265 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 10:24:13.153108 31433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:24:13.152170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.21.33.190:36871)
0929 10:24:13.152326 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:13.152331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:13.152563 (+ 232us) server_negotiation.cc:407] Connection header received
0929 10:24:13.152626 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:13.152629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:13.152663 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:13.152737 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:13.152959 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50498: BlockingRecv error: recv got EOF from 127.0.0.1:50498 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:24:13.153301 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31454
I20260929 10:24:13.158666 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31585
I20260929 10:24:13.163662 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31716
I20260929 10:24:13.170168 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31363
2026-09-29T10:24:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (822 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:24:13 dist-test-slave-2rbl krb5kdc[31854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:24:13 dist-test-slave-2rbl krb5kdc[31854](info): set up 2 sockets
Sep 29 10:24:13 dist-test-slave-2rbl krb5kdc[31854](info): commencing operation
krb5kdc: starting...
W20260929 10:24:14.214847 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:24:14 dist-test-slave-2rbl krb5kdc[31854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:24:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:24:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:24:14.338522 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:45489
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:45489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:24:14.449946 31867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:14.450213 31867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:14.450253 31867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:14.454622 31867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:24:14.454687 31867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:14.454710 31867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:14.454761 31867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:24:14.454804 31867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:24:14.460214 31867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:45489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:45489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:14.461288 31867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:14.462316 31867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:14.470476 31875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:14.470476 31873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:14.470485 31872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:14.470774 31867 server_base.cc:1034] running on GCE node
I20260929 10:24:14.471336 31867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:14.472443 31867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:14.473649 31867 hybrid_clock.cc:648] HybridClock initialized: now 1790677454473637 us; error 26 us; skew 500 ppm
Sep 29 10:24:14 dist-test-slave-2rbl krb5kdc[31854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677454, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:14.476423 31867 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:24:14.477285 31867 webserver.cc:460] Webserver started at http://127.21.33.190:35467/ using document root <none> and password file <none>
I20260929 10:24:14.477792 31867 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:14.477849 31867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:14.478005 31867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:14.479926 31867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "707423f5705b42739d6fa0ff8fdf2594"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "9900e092b3900b278f74f7c38324d918"
server_key_iv: "710e39952fba6d17d738861c0728f4ca"
server_key_version: "encryptionkey@0"
I20260929 10:24:14.480425 31867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "707423f5705b42739d6fa0ff8fdf2594"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "9900e092b3900b278f74f7c38324d918"
server_key_iv: "710e39952fba6d17d738861c0728f4ca"
server_key_version: "encryptionkey@0"
I20260929 10:24:14.483980 31867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:24:14.486392 31882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:14.487706 31867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:24:14.487841 31867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "707423f5705b42739d6fa0ff8fdf2594"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "9900e092b3900b278f74f7c38324d918"
server_key_iv: "710e39952fba6d17d738861c0728f4ca"
server_key_version: "encryptionkey@0"
I20260929 10:24:14.487949 31867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:14.508097 31867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:14.511839 31867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:14.512059 31867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:14.519657 31867 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:45489
I20260929 10:24:14.519678 31934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:45489 every 8 connection(s)
I20260929 10:24:14.520629 31867 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:24:14.523658 31935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:14.524945 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31867
I20260929 10:24:14.525120 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:24:14.525451 21638 external_mini_cluster.cc:1428] Setting key b32acab899ba210da55edde9a90ef332
I20260929 10:24:14.530000 31935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594: Bootstrap starting.
I20260929 10:24:14.532210 31935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:14.532946 31935 log.cc:824] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:14.534855 31935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594: No bootstrap required, opened a new log
Sep 29 10:24:14 dist-test-slave-2rbl krb5kdc[31854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:24:14.538072 31935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707423f5705b42739d6fa0ff8fdf2594" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45489 } }
I20260929 10:24:14.538347 31935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:14.538406 31935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707423f5705b42739d6fa0ff8fdf2594, State: Initialized, Role: FOLLOWER
I20260929 10:24:14.538831 31935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707423f5705b42739d6fa0ff8fdf2594" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45489 } }
I20260929 10:24:14.538981 31935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:24:14.539068 31935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:24:14.539178 31935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:24:14.540292 31935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707423f5705b42739d6fa0ff8fdf2594" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45489 } }
I20260929 10:24:14.540653 31935 leader_election.cc:302] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707423f5705b42739d6fa0ff8fdf2594; no voters:
I20260929 10:24:14.541019 31935 leader_election.cc:288] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:24:14.541226 31940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:24:14.541508 31940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [term 1 LEADER]: Becoming Leader. State: Replica: 707423f5705b42739d6fa0ff8fdf2594, State: Running, Role: LEADER
I20260929 10:24:14.541888 31940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707423f5705b42739d6fa0ff8fdf2594" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45489 } }
I20260929 10:24:14.542402 31935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:24:14.542665 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:14.526967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.21.33.190:45489)
0929 10:24:14.527441 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:14.527458 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:14.527499 (+ 41us) server_negotiation.cc:407] Connection header received
0929 10:24:14.528210 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:14.528229 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:14.528604 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:14.528975 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:14.529971 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:14.531136 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:14.532128 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:14.532734 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:14.532928 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:14.535970 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:14.535976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:14.535990 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:14.536020 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:14.539004 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:14.539598 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:14.539603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:14.539607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:14.539737 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:14.540061 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:14.540063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:14.540065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:14.540549 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:14.540728 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:14.541065 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 10:24:14.541560 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
I20260929 10:24:14.543659 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707423f5705b42739d6fa0ff8fdf2594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707423f5705b42739d6fa0ff8fdf2594" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45489 } } }
I20260929 10:24:14.543804 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [sys.catalog]: This master's current role is: LEADER
I20260929 10:24:14.544106 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707423f5705b42739d6fa0ff8fdf2594. Latest consensus state: current_term: 1 leader_uuid: "707423f5705b42739d6fa0ff8fdf2594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707423f5705b42739d6fa0ff8fdf2594" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 45489 } } }
I20260929 10:24:14.544209 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594 [sys.catalog]: This master's current role is: LEADER
W20260929 10:24:14.547928 31955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:24:14.548007 31955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:24:14.548141 31950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:24:14.548933 31950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:24:14.554589 31950 catalog_manager.cc:1347] Generated new cluster ID: ee919f120a0249ffac23358200c80d77
I20260929 10:24:14.554665 31950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:24:14.573865 31950 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:24:14.575147 31950 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:24:14.584841 31950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594: Generated new TSK 0
I20260929 10:24:14.585665 31950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:24:14.622145 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:45489
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:24:14.736354 31961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:14.736693 31961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:14.736779 31961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:14.741516 31961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:14.741621 31961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:14.741729 31961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:24:14.747694 31961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:45489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:14.749006 31961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:14.750277 31961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:14.760393 31969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:14.760457 31966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:14.760396 31967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:14.760550 31961 server_base.cc:1034] running on GCE node
I20260929 10:24:14.761101 31961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:14.761756 31961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:14.762918 31961 hybrid_clock.cc:648] HybridClock initialized: now 1790677454762902 us; error 29 us; skew 500 ppm
Sep 29 10:24:14 dist-test-slave-2rbl krb5kdc[31854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677454, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:14.765846 31961 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:24:14.766726 31961 webserver.cc:460] Webserver started at http://127.21.33.129:42001/ using document root <none> and password file <none>
I20260929 10:24:14.767271 31961 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:14.767361 31961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:14.767594 31961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:14.769688 31961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "457f3499f6ad4bd194ed9e6012bd926b"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "d46d10a762e84ae81241ca5f080efe25"
server_key_iv: "495529a43a06423f25803975a9d60db8"
server_key_version: "encryptionkey@0"
I20260929 10:24:14.770259 31961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "457f3499f6ad4bd194ed9e6012bd926b"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "d46d10a762e84ae81241ca5f080efe25"
server_key_iv: "495529a43a06423f25803975a9d60db8"
server_key_version: "encryptionkey@0"
I20260929 10:24:14.774008 31961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:24:14.776351 31976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:14.777640 31961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:24:14.777773 31961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "457f3499f6ad4bd194ed9e6012bd926b"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "d46d10a762e84ae81241ca5f080efe25"
server_key_iv: "495529a43a06423f25803975a9d60db8"
server_key_version: "encryptionkey@0"
I20260929 10:24:14.777886 31961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:14.794827 31961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:14.798084 31961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:14.798292 31961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:14.798931 31961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:14.799727 31961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:14.799796 31961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:14.799866 31961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:14.799918 31961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:14.811185 31961 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:41383
I20260929 10:24:14.811468 32089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:41383 every 8 connection(s)
I20260929 10:24:14.812419 31961 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:24:14.819818 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 31961
I20260929 10:24:14.819962 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:24:14.820273 21638 external_mini_cluster.cc:1428] Setting key fe473a8d48c260c2386be0752224d40f
Sep 29 10:24:14 dist-test-slave-2rbl krb5kdc[31854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677454, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:24:14.831300 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:14.814328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:57111 (local address 127.21.33.190:45489)
0929 10:24:14.814454 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:14.814458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:14.815748 (+ 1290us) server_negotiation.cc:407] Connection header received
0929 10:24:14.817757 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:14.817762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:14.817834 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:14.817939 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:14.820076 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:14.820851 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:14.822501 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:14.822926 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:14.823020 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:14.826359 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:14.826363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:14.826366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:14.826400 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:14.828443 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:14.829248 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:14.829254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:14.829256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:14.829359 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:14.829843 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:14.829847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:14.829849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:14.830106 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:14.830203 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:14.830956 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 10:24:14.831110 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 10:24:14.832235 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:14.814836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:45489 (local address 127.21.33.129:57111)
0929 10:24:14.815576 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:14.815645 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:14.817520 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:14.818347 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:14.818358 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:14.818961 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:14.819867 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:14.819890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:14.821111 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:14.821117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:14.822352 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:14.822366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:14.823240 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:14.823246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:14.823482 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:14.824210 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:14.824230 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:14.826140 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:14.828634 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:14.828680 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:14.828698 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:14.829102 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:14.829467 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:14.829471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:14.829473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:14.829686 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:14.830274 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:14.830281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:14.830627 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 10:24:14.830864 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 10:24:14.831216 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":156,"threads_started":1}
I20260929 10:24:14.833673 32090 heartbeater.cc:347] Connected to a master server at 127.21.33.190:45489
I20260929 10:24:14.833947 32090 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:24:14.834749 32090 heartbeater.cc:511] Master 127.21.33.190:45489 requested a full tablet report, sending...
I20260929 10:24:14.837558 31899 ts_manager.cc:194] Registered new tserver with Master: 457f3499f6ad4bd194ed9e6012bd926b (127.21.33.129:41383)
I20260929 10:24:14.838903 31899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:57111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:24:14.854465 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:45489
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:24:14.966565 32095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:14.966830 32095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:14.966871 32095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:14.970903 32095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:14.970955 32095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:14.971027 32095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:24:14.976263 32095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:45489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:14.977260 32095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:14.978231 32095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:14.988010 32100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:14.987986 32101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:14.988289 32103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:14.988425 32095 server_base.cc:1034] running on GCE node
I20260929 10:24:14.988780 32095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:14.989485 32095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:14.990677 32095 hybrid_clock.cc:648] HybridClock initialized: now 1790677454990658 us; error 30 us; skew 500 ppm
Sep 29 10:24:14 dist-test-slave-2rbl krb5kdc[31854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677454, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:14.993687 32095 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:24:14.994602 32095 webserver.cc:460] Webserver started at http://127.21.33.130:40041/ using document root <none> and password file <none>
I20260929 10:24:14.995139 32095 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:14.995199 32095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:14.995361 32095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:14.997406 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "e870ec36bba541f19dc013229222e287"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "2e19ebb34709c58aed76106df1d60c95"
server_key_iv: "2003d736e04076a7ce2f8b70fc8d6a81"
server_key_version: "encryptionkey@0"
I20260929 10:24:14.997928 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "e870ec36bba541f19dc013229222e287"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "2e19ebb34709c58aed76106df1d60c95"
server_key_iv: "2003d736e04076a7ce2f8b70fc8d6a81"
server_key_version: "encryptionkey@0"
I20260929 10:24:15.001670 32095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:24:15.004000 32110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:15.005298 32095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:24:15.005386 32095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "e870ec36bba541f19dc013229222e287"
format_stamp: "Formatted at 2026-09-29 10:24:14 on dist-test-slave-2rbl"
server_key: "2e19ebb34709c58aed76106df1d60c95"
server_key_iv: "2003d736e04076a7ce2f8b70fc8d6a81"
server_key_version: "encryptionkey@0"
I20260929 10:24:15.005467 32095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:15.019598 32095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:15.022457 32095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:15.022603 32095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:15.023135 32095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:15.023913 32095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:15.023958 32095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:15.023996 32095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:15.024012 32095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:15.033105 32095 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:40645
I20260929 10:24:15.033141 32223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:40645 every 8 connection(s)
I20260929 10:24:15.034009 32095 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:24:15.041205 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 32095
I20260929 10:24:15.041383 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:24:15.041689 21638 external_mini_cluster.cc:1428] Setting key 0433c1996d23efa0c75c3a47dbfc26bf
Sep 29 10:24:15 dist-test-slave-2rbl krb5kdc[31854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677454, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:24:15.049437 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.035864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:54075 (local address 127.21.33.190:45489)
0929 10:24:15.036017 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.036022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.037086 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 10:24:15.037999 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.038002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.038047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.038121 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:15.039558 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.040084 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.041309 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.041643 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.041764 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.045138 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:15.045145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:15.045150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:15.045182 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:15.046994 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:15.047546 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:15.047552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:15.047557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:15.047661 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:15.048097 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:15.048100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:15.048102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:15.048352 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:15.048465 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.049023 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.049169 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:24:15.050184 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.036299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:45489 (local address 127.21.33.130:54075)
0929 10:24:15.036937 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.036979 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:15.037800 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:15.038311 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:15.038319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:15.038826 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:15.039424 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.039439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.040199 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.040201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.041114 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.041133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.041823 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.041828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.042052 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.042810 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:15.042833 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:15.044919 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:15.047139 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:15.047144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:15.047158 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:15.047439 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:15.047790 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:15.047794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:15.047796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:15.047992 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:15.048503 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:15.048509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:15.048761 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 10:24:15.048932 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 10:24:15.049276 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":105,"threads_started":1}
I20260929 10:24:15.051791 32224 heartbeater.cc:347] Connected to a master server at 127.21.33.190:45489
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:24:15.052042 32224 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:15.052532 32224 heartbeater.cc:511] Master 127.21.33.190:45489 requested a full tablet report, sending...
I20260929 10:24:15.053701 31899 ts_manager.cc:194] Registered new tserver with Master: e870ec36bba541f19dc013229222e287 (127.21.33.130:40645)
I20260929 10:24:15.054359 31899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:54075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:24:15.070302 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:45489
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:24:15.186169 32229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:15.186411 32229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:15.186455 32229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:15.190647 32229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:15.190724 32229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:15.190809 32229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:24:15.197680 32229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:45489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:15.198812 32229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:15.199867 32229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:15.210233 32235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:15.210242 32234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:15.210275 32237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:15.210701 32229 server_base.cc:1034] running on GCE node
I20260929 10:24:15.211169 32229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:15.211814 32229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:15.213004 32229 hybrid_clock.cc:648] HybridClock initialized: now 1790677455212987 us; error 29 us; skew 500 ppm
Sep 29 10:24:15 dist-test-slave-2rbl krb5kdc[31854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677455, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:15.222028 32229 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:24:15.223002 32229 webserver.cc:460] Webserver started at http://127.21.33.131:42141/ using document root <none> and password file <none>
I20260929 10:24:15.223623 32229 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:15.223721 32229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:15.223958 32229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:24:15.226184 32229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "d1631fca90c247c790138a1e74db92cc"
format_stamp: "Formatted at 2026-09-29 10:24:15 on dist-test-slave-2rbl"
server_key: "a31a86f5380b018714e6972f1b1cae4b"
server_key_iv: "b40208993ba861ffc7e44d9033ed1b78"
server_key_version: "encryptionkey@0"
I20260929 10:24:15.226761 32229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "d1631fca90c247c790138a1e74db92cc"
format_stamp: "Formatted at 2026-09-29 10:24:15 on dist-test-slave-2rbl"
server_key: "a31a86f5380b018714e6972f1b1cae4b"
server_key_iv: "b40208993ba861ffc7e44d9033ed1b78"
server_key_version: "encryptionkey@0"
I20260929 10:24:15.230564 32229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:24:15.233064 32244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:15.234344 32229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:24:15.234484 32229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "d1631fca90c247c790138a1e74db92cc"
format_stamp: "Formatted at 2026-09-29 10:24:15 on dist-test-slave-2rbl"
server_key: "a31a86f5380b018714e6972f1b1cae4b"
server_key_iv: "b40208993ba861ffc7e44d9033ed1b78"
server_key_version: "encryptionkey@0"
I20260929 10:24:15.234609 32229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:15.255137 32229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:15.258492 32229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:15.258729 32229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:15.259495 32229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:15.260427 32229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:15.260502 32229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:15.260572 32229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:15.260651 32229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:15.270051 32229 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:39433
I20260929 10:24:15.270078 32357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:39433 every 8 connection(s)
I20260929 10:24:15.270949 32229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:24:15.276818 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 32229
I20260929 10:24:15.276963 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:24:15.277299 21638 external_mini_cluster.cc:1428] Setting key 8930acdf12212bad3eccbd0531368461
Sep 29 10:24:15 dist-test-slave-2rbl krb5kdc[31854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677455, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:24:15.284960 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.272776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:42671 (local address 127.21.33.190:45489)
0929 10:24:15.272946 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.272950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.273858 (+ 908us) server_negotiation.cc:407] Connection header received
0929 10:24:15.275158 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.275161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.275209 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.275284 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:15.276427 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.277245 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.278813 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.279066 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.279158 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.281772 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:15.281775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:15.281777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:15.281801 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:15.283134 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:15.283608 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:15.283611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:15.283612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:15.283684 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:15.283977 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:15.283980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:15.283981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:15.284155 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:15.284231 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.284669 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.284801 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:24:15.285792 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.273243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:45489 (local address 127.21.33.131:42671)
0929 10:24:15.273712 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.273752 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:15.274956 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:15.275443 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:15.275449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:15.275799 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:15.276294 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.276307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.277389 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.277395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.278683 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.278696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.279226 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.279230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.279418 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.280066 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:15.280083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:15.281611 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:15.283233 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:15.283238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:15.283249 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:15.283492 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:15.283763 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:15.283765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:15.283766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:15.283901 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:15.284239 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:15.284245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:15.284467 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 10:24:15.284623 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 10:24:15.284867 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260929 10:24:15.287267 32358 heartbeater.cc:347] Connected to a master server at 127.21.33.190:45489
I20260929 10:24:15.287524 32358 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:15.288003 32358 heartbeater.cc:511] Master 127.21.33.190:45489 requested a full tablet report, sending...
I20260929 10:24:15.289114 31899 ts_manager.cc:194] Registered new tserver with Master: d1631fca90c247c790138a1e74db92cc (127.21.33.131:39433)
I20260929 10:24:15.289726 31899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:42671
I20260929 10:24:15.291527 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:24:15.300869 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.293723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.21.33.190:45489)
0929 10:24:15.293845 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.293849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.293993 (+ 144us) server_negotiation.cc:407] Connection header received
0929 10:24:15.294080 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.294082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.294120 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.294182 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:15.294818 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.295318 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.296179 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.296391 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.296482 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.297888 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:15.297891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:15.297894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:15.297924 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:15.299404 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:15.299792 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:15.299795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:15.299796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:15.299873 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:15.300099 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:15.300102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:15.300104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:15.300317 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:15.300421 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.300653 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.300746 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:24:15.303679 31899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:24:15.306735 31899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:24:15.321074 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.315362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:39433 (local address 127.0.0.1:39758)
0929 10:24:15.315948 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.315999 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:15.316122 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:15.316858 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:15.316864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:15.316886 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:15.317026 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.317033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.318266 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.318272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.319591 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.319601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.320524 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.320528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.320633 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.320683 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 10:24:15.320797 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 10:24:15.320918 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":27,"thread_start_us":98,"threads_started":1}
I20260929 10:24:15.321079 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.315573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.21.33.131:39433)
0929 10:24:15.316302 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.316314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.316341 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:24:15.316423 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.316432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.316669 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.317175 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:15.317356 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.318137 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.319721 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.320397 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.320521 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.320671 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.320831 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.320923 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":95,"threads_started":1}
I20260929 10:24:15.321544 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.315915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:40645 (local address 127.0.0.1:37992)
0929 10:24:15.316367 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.316379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:15.316449 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:15.317020 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:15.317023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:15.317037 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:15.317145 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.317150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.318669 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.318672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.319810 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.319817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.321081 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.321085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.321229 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.321251 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:24:15.321318 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 10:24:15.321409 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260929 10:24:15.321882 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.315266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.21.33.129:41383)
0929 10:24:15.315823 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.315845 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.316771 (+ 926us) server_negotiation.cc:407] Connection header received
0929 10:24:15.316959 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.316972 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.317119 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.317340 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:15.317528 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.318596 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.320027 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.320782 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.320889 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.321017 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.321654 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.321750 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":132,"threads_started":1}
I20260929 10:24:15.322031 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.316066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.21.33.130:40645)
0929 10:24:15.316393 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.316404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.316442 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:24:15.316572 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.316577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.316721 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.316865 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:15.317299 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.318510 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.319921 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.320782 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.320889 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.321017 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.321781 (+ 764us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.321875 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
I20260929 10:24:15.324469 32292 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec73eaf00aa401783a3162cbbc2dadd (DEFAULT_TABLE table=test-table [id=6349fc1af4454e3f920724116f8b22b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:24:15.324469 32158 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec73eaf00aa401783a3162cbbc2dadd (DEFAULT_TABLE table=test-table [id=6349fc1af4454e3f920724116f8b22b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:24:15.325248 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.315169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:41383 (local address 127.0.0.1:35514)
0929 10:24:15.315937 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.316699 (+ 762us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:15.316841 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:15.317333 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:15.317336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:15.317354 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:15.317433 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.317438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.318710 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.318713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.319902 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.319912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.321081 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.321085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.321506 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.321526 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 10:24:15.321780 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 10:24:15.321849 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":72,"thread_start_us":98,"threads_started":1}
I20260929 10:24:15.325654 32158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec73eaf00aa401783a3162cbbc2dadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:15.325657 32292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec73eaf00aa401783a3162cbbc2dadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:15.327739 32024 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec73eaf00aa401783a3162cbbc2dadd (DEFAULT_TABLE table=test-table [id=6349fc1af4454e3f920724116f8b22b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:24:15.328611 32024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec73eaf00aa401783a3162cbbc2dadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:15.333004 32375 tablet_bootstrap.cc:492] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Bootstrap starting.
I20260929 10:24:15.333493 32376 tablet_bootstrap.cc:492] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Bootstrap starting.
I20260929 10:24:15.335007 32377 tablet_bootstrap.cc:492] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: Bootstrap starting.
I20260929 10:24:15.336300 32376 tablet_bootstrap.cc:654] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:15.336971 32377 tablet_bootstrap.cc:654] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:15.337162 32376 log.cc:824] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:15.337495 32375 tablet_bootstrap.cc:654] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:15.337757 32377 log.cc:824] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:15.338721 32375 log.cc:824] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:15.339844 32377 tablet_bootstrap.cc:492] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: No bootstrap required, opened a new log
I20260929 10:24:15.340041 32377 ts_tablet_manager.cc:1397] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:24:15.340278 32376 tablet_bootstrap.cc:492] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: No bootstrap required, opened a new log
I20260929 10:24:15.340453 32376 ts_tablet_manager.cc:1397] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 10:24:15.340639 32375 tablet_bootstrap.cc:492] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: No bootstrap required, opened a new log
I20260929 10:24:15.340824 32375 ts_tablet_manager.cc:1397] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260929 10:24:15.344323 32375 raft_consensus.cc:348] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.344360 32377 raft_consensus.cc:348] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.344359 32376 raft_consensus.cc:348] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.344559 32375 raft_consensus.cc:374] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:15.344606 32377 raft_consensus.cc:374] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:15.344655 32375 raft_consensus.cc:729] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1631fca90c247c790138a1e74db92cc, State: Initialized, Role: FOLLOWER
I20260929 10:24:15.344658 32377 raft_consensus.cc:729] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457f3499f6ad4bd194ed9e6012bd926b, State: Initialized, Role: FOLLOWER
I20260929 10:24:15.344669 32376 raft_consensus.cc:374] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:15.344719 32376 raft_consensus.cc:729] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e870ec36bba541f19dc013229222e287, State: Initialized, Role: FOLLOWER
I20260929 10:24:15.345100 32375 consensus_queue.cc:260] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.345130 32377 consensus_queue.cc:260] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.345130 32376 consensus_queue.cc:260] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.345745 32358 heartbeater.cc:503] Master 127.21.33.190:45489 was elected leader, sending a full tablet report...
I20260929 10:24:15.346048 32375 ts_tablet_manager.cc:1428] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260929 10:24:15.346069 32377 ts_tablet_manager.cc:1428] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:24:15.346405 32090 heartbeater.cc:503] Master 127.21.33.190:45489 was elected leader, sending a full tablet report...
I20260929 10:24:15.347155 32376 ts_tablet_manager.cc:1428] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:24:15.347479 32224 heartbeater.cc:503] Master 127.21.33.190:45489 was elected leader, sending a full tablet report...
W20260929 10:24:15.521924 32359 tablet.cc:2369] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:24:15.535187 32225 tablet.cc:2369] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:24:15.547170 32383 raft_consensus.cc:482] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:24:15.547418 32383 raft_consensus.cc:504] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.548664 32383 leader_election.cc:288] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 457f3499f6ad4bd194ed9e6012bd926b (127.21.33.129:41383), d1631fca90c247c790138a1e74db92cc (127.21.33.131:39433)
I20260929 10:24:15.553745 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.549020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:57821 (local address 127.21.33.129:41383)
0929 10:24:15.549158 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.549164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.549239 (+ 75us) server_negotiation.cc:407] Connection header received
0929 10:24:15.549306 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.549310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.549352 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.549411 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:15.549777 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.550607 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.552446 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.553042 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.553178 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.553247 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.553546 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.553616 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:24:15.553743 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.549195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:39621 (local address 127.21.33.131:39433)
0929 10:24:15.549347 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.549352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.549464 (+ 112us) server_negotiation.cc:407] Connection header received
0929 10:24:15.549586 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.549588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.549628 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.549692 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:15.549962 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.550674 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.552445 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.553059 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.553170 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.553247 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.553546 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.553616 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:24:15.553788 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.548854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:41383 (local address 127.21.33.130:57821)
0929 10:24:15.549021 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.549037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:15.549143 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:15.549437 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:15.549440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:15.549478 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:15.549626 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.549632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.550739 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.550745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.552298 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.552310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.553177 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.553181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.553413 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.553447 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 10:24:15.553517 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 10:24:15.553612 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":215,"spinlock_wait_cycles":5376}
I20260929 10:24:15.553797 32384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.548994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:39433 (local address 127.21.33.130:39621)
0929 10:24:15.549347 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.549361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:15.549480 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:15.549740 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:15.549746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:15.549767 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:15.549864 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.549868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.550771 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.550774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.552303 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:15.552310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.553177 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.553181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:15.553395 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.553435 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:24:15.553517 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:24:15.553612 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":244,"spinlock_wait_cycles":22144,"thread_start_us":103,"threads_started":1}
I20260929 10:24:15.554533 32044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec73eaf00aa401783a3162cbbc2dadd" candidate_uuid: "e870ec36bba541f19dc013229222e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457f3499f6ad4bd194ed9e6012bd926b" is_pre_election: true
I20260929 10:24:15.554555 32312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec73eaf00aa401783a3162cbbc2dadd" candidate_uuid: "e870ec36bba541f19dc013229222e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1631fca90c247c790138a1e74db92cc" is_pre_election: true
I20260929 10:24:15.554842 32044 raft_consensus.cc:2454] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e870ec36bba541f19dc013229222e287 in term 0.
I20260929 10:24:15.554848 32312 raft_consensus.cc:2454] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e870ec36bba541f19dc013229222e287 in term 0.
I20260929 10:24:15.555328 32113 leader_election.cc:302] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 457f3499f6ad4bd194ed9e6012bd926b, d1631fca90c247c790138a1e74db92cc, e870ec36bba541f19dc013229222e287; no voters:
I20260929 10:24:15.555603 32383 raft_consensus.cc:2781] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:24:15.555686 32383 raft_consensus.cc:482] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:24:15.555732 32383 raft_consensus.cc:3037] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:24:15.556712 32383 raft_consensus.cc:504] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.557108 32383 leader_election.cc:288] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [CANDIDATE]: Term 1 election: Requested vote from peers 457f3499f6ad4bd194ed9e6012bd926b (127.21.33.129:41383), d1631fca90c247c790138a1e74db92cc (127.21.33.131:39433)
I20260929 10:24:15.557413 32044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec73eaf00aa401783a3162cbbc2dadd" candidate_uuid: "e870ec36bba541f19dc013229222e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457f3499f6ad4bd194ed9e6012bd926b"
I20260929 10:24:15.557512 32312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec73eaf00aa401783a3162cbbc2dadd" candidate_uuid: "e870ec36bba541f19dc013229222e287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1631fca90c247c790138a1e74db92cc"
I20260929 10:24:15.557540 32044 raft_consensus.cc:3037] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:24:15.557675 32312 raft_consensus.cc:3037] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:24:15.558660 32044 raft_consensus.cc:2454] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e870ec36bba541f19dc013229222e287 in term 1.
I20260929 10:24:15.558688 32312 raft_consensus.cc:2454] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e870ec36bba541f19dc013229222e287 in term 1.
I20260929 10:24:15.558935 32113 leader_election.cc:302] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 457f3499f6ad4bd194ed9e6012bd926b, e870ec36bba541f19dc013229222e287; no voters:
I20260929 10:24:15.559127 32383 raft_consensus.cc:2781] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:24:15.559401 32383 raft_consensus.cc:686] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 1 LEADER]: Becoming Leader. State: Replica: e870ec36bba541f19dc013229222e287, State: Running, Role: LEADER
I20260929 10:24:15.559744 32383 consensus_queue.cc:237] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } }
I20260929 10:24:15.563017 31899 catalog_manager.cc:5495] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 reported cstate change: term changed from 0 to 1, leader changed from <none> to e870ec36bba541f19dc013229222e287 (127.21.33.130). New cstate: current_term: 1 leader_uuid: "e870ec36bba541f19dc013229222e287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e870ec36bba541f19dc013229222e287" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 } health_report { overall_health: UNKNOWN } } }
W20260929 10:24:15.563858 32091 tablet.cc:2369] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:24:15.613250 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.609441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.21.33.130:40645)
0929 10:24:15.609583 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:15.609589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:15.609621 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:24:15.609837 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:15.609840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:15.609895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:15.610017 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:24:15.610288 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.610823 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.611693 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:15.611923 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:15.612031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:15.612367 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:24:15.612819 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:24:15.612960 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:15.612997 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 10:24:15.613054 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 10:24:15.619360 32312 raft_consensus.cc:1261] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 1 FOLLOWER]: Refusing update from remote peer e870ec36bba541f19dc013229222e287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:24:15.619406 32044 raft_consensus.cc:1261] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 1 FOLLOWER]: Refusing update from remote peer e870ec36bba541f19dc013229222e287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:24:15.620126 32383 consensus_queue.cc:1035] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "457f3499f6ad4bd194ed9e6012bd926b" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:24:15.620339 32385 consensus_queue.cc:1035] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1631fca90c247c790138a1e74db92cc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:24:15.628747 32390 mvcc.cc:205] Tried to move back new op lower bound from 7334614858209185792 to 7334614857975656448. Current Snapshot: MvccSnapshot[applied={T|T < 7334614858209185792}]
I20260929 10:24:15.630532 32389 mvcc.cc:205] Tried to move back new op lower bound from 7334614858209185792 to 7334614857975656448. Current Snapshot: MvccSnapshot[applied={T|T < 7334614858209185792}]
I20260929 10:24:15.637557 31899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:24:15.637756 31899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:24:15.640367 31899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 707423f5705b42739d6fa0ff8fdf2594: Sending DeleteTablet for 3 replicas of tablet 6ec73eaf00aa401783a3162cbbc2dadd
I20260929 10:24:15.641247 32158 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec73eaf00aa401783a3162cbbc2dadd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:24:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:37992
I20260929 10:24:15.641242 32292 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec73eaf00aa401783a3162cbbc2dadd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:24:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:39758
I20260929 10:24:15.641395 32024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec73eaf00aa401783a3162cbbc2dadd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:24:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:35514
I20260929 10:24:15.641788 32399 tablet_replica.cc:330] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: stopping tablet replica
I20260929 10:24:15.641808 32400 tablet_replica.cc:330] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b: stopping tablet replica
I20260929 10:24:15.642140 32400 raft_consensus.cc:2229] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:24:15.642225 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31961
I20260929 10:24:15.642150 32399 raft_consensus.cc:2229] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:24:15.642551 32400 raft_consensus.cc:2258] T 6ec73eaf00aa401783a3162cbbc2dadd P 457f3499f6ad4bd194ed9e6012bd926b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:24:15.642760 32399 raft_consensus.cc:2258] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:24:15.643136 32398 tablet_replica.cc:330] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: stopping tablet replica
I20260929 10:24:15.643579 32398 raft_consensus.cc:2229] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:24:15.643886 32398 raft_consensus.cc:2258] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:24:15.644850 32399 ts_tablet_manager.cc:1905] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:24:15.645383 32398 ts_tablet_manager.cc:1905] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:24:15.647997 32398 ts_tablet_manager.cc:1918] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:24:15.648108 32398 log.cc:1196] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/6ec73eaf00aa401783a3162cbbc2dadd
I20260929 10:24:15.648447 32398 ts_tablet_manager.cc:1939] T 6ec73eaf00aa401783a3162cbbc2dadd P d1631fca90c247c790138a1e74db92cc: Deleting consensus metadata
I20260929 10:24:15.648466 32399 ts_tablet_manager.cc:1918] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:24:15.648690 32399 log.cc:1196] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/6ec73eaf00aa401783a3162cbbc2dadd
I20260929 10:24:15.649042 32399 ts_tablet_manager.cc:1939] T 6ec73eaf00aa401783a3162cbbc2dadd P e870ec36bba541f19dc013229222e287: Deleting consensus metadata
I20260929 10:24:15.649721 31886 catalog_manager.cc:4841] TS d1631fca90c247c790138a1e74db92cc (127.21.33.131:39433): tablet 6ec73eaf00aa401783a3162cbbc2dadd (table test-table [id=6349fc1af4454e3f920724116f8b22b0]) successfully deleted
I20260929 10:24:15.650295 31885 catalog_manager.cc:4841] TS e870ec36bba541f19dc013229222e287 (127.21.33.130:40645): tablet 6ec73eaf00aa401783a3162cbbc2dadd (table test-table [id=6349fc1af4454e3f920724116f8b22b0]) successfully deleted
W20260929 10:24:15.652175 31885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:41383 (error 108)
I20260929 10:24:15.652256 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 32095
I20260929 10:24:15.653995 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:15.653740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:41383 (local address 127.0.0.1:35520)
0929 10:24:15.653866 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:15.653923 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:41383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 10:24:15.654147 31885 catalog_manager.cc:4584] TS 457f3499f6ad4bd194ed9e6012bd926b (127.21.33.129:41383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ec73eaf00aa401783a3162cbbc2dadd: Network error: Client connection negotiation failed: client connection to 127.21.33.129:41383: connect: Connection refused (error 111)
I20260929 10:24:15.657945 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 32229
I20260929 10:24:15.663120 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 31867
2026-09-29T10:24:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2494 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:24:15 dist-test-slave-2rbl krb5kdc[32404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:24:15 dist-test-slave-2rbl krb5kdc[32404](info): set up 2 sockets
Sep 29 10:24:15 dist-test-slave-2rbl krb5kdc[32404](info): commencing operation
krb5kdc: starting...
W20260929 10:24:16.698728 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:24:16 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:24:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:24:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/rangeradmin_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/rangeradmin_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/rangerlookup_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/rangerlookup_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
I20260929 10:24:16.872006 21638 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksWWfjY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/postgres -l logfile start
2026-09-29 10:24:19.445 UTC [32434] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 10:24:19.445 UTC [32434] LOG: listening on IPv4 address "127.21.33.148", port 44923
2026-09-29 10:24:19.449 UTC [32434] LOG: listening on Unix socket "/tmp/.s.PGSQL.44923"
2026-09-29 10:24:19.470 UTC [32437] LOG: database system was shut down at 2026-09-29 10:24:17 UTC
2026-09-29 10:24:19.474 UTC [32434] LOG: database system is ready to accept connections
I20260929 10:24:20.430485 21638 mini_postgres.cc:96] Postgres bound to 44923
2026-09-29 10:24:20.437 UTC [32445] FATAL: database "slave" does not exist
127.21.33.148:44923 - accepting connections
I20260929 10:24:20.438020 21638 mini_ranger.cc:161] Starting Ranger...
I20260929 10:24:20.453770 21638 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 10:24:20.938634 21638 mini_ranger.cc:87] Created ranger Postgres database
I20260929 10:24:20.938736 21638 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-admin
2026-09-29 10:24:21,212 [I] DB FLAVOR :POSTGRES
2026-09-29 10:24:21,213 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:24:21,213 [I] Checking connection..
2026-09-29 10:24:21,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 10:24:21,451 [I] Checking connection passed.
2026-09-29 10:24:21,451 [I] --------- Verifying version history table ---------
2026-09-29 10:24:21,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:24:21,707 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:24:21,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:24:21,954 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:24:21,954 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 10:24:21,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 10:24:22.204 UTC [32509] WARNING: there is no transaction in progress
2026-09-29 10:24:22,211 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 10:24:22,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:24:22,468 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 10:24:22,468 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:24:22,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 10:24:22,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 10:24:22,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2rbl.c.gcp-upstream.internal','N') ;"
2026-09-29 10:24:23,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:24:23,435 [I] Table x_portal_user does not exist in database ranger
2026-09-29 10:24:23,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:24:23,681 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 10:24:23,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:24:23,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:24:24,141 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 10:24:24,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 10:24:25.126 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.144 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.161 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.179 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.197 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.218 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.275 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.282 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.297 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.307 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.318 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.332 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.341 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.348 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.355 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:25.362 UTC [32617] WARNING: there is no transaction in progress
2026-09-29 10:24:26,021 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 10:24:26,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 10:24:26,244 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 10:24:26,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:24:26,477 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 10:24:26,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:24:26,707 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 10:24:26,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:24:26,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:24:27,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2rbl.c.gcp-upstream.internal';"
2026-09-29 10:24:27,435 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 10:24:27,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 10:24:27,675 [I] DB_PATCHES have already been applied
I20260929 10:24:27.680485 21638 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksWWfjY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 10:24:27.680583 21638 mini_ranger.cc:193] Using host: 127.21.33.148
I20260929 10:24:27.684090 21638 mini_ranger.cc:227] Ranger admin URL: http://127.21.33.148:59675
Sep 29, 2026 10:24:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:24:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:24:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:24:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:24:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:24:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:24:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 10:24:28.787878 21638 mini_ranger.cc:160] Time spent starting Ranger: real 8.350s user 0.000s sys 0.010s
Sep 29, 2026 10:24:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:24:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 10:24:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:24:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:24:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 10:24:49.495991 21638 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/rangerkms_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/rangerkms_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.33.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.33.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 10:24:49 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677489, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 10:24:49.597007 21638 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 10:24:49.606812 21638 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 10:24:50.553718 21638 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 10:24:50.553819 21638 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms
I20260929 10:24:50.553922 21638 mini_ranger_kms.cc:227] Using postgres at 127.21.33.148:44923
2026-09-29 10:24:50,732 [I] DB FLAVOR :POSTGRES
2026-09-29 10:24:50,732 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:24:50,732 [I] Checking connection
2026-09-29 10:24:50,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 10:24:50,980 [I] connection success
2026-09-29 10:24:50,981 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 10:24:50,981 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 10:24:50,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 10:24:51,224 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 10:24:51,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:24:51,224 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 10:24:51,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:44923/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 10:24:51,472 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 10:24:51.933111 21638 mini_ranger_kms.cc:310] Created kms service
I20260929 10:24:52.158206 21638 mini_ranger_kms.cc:326] Created kudu user
I20260929 10:24:52.201092 21638 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 10:24:52 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677489, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:24:52.584714 21638 mini_ranger_kms.cc:384] Added ranger policy
I20260929 10:24:52.584872 21638 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 10:24:52.584897 21638 mini_ranger_kms.cc:240] Using host: 127.21.33.148
I20260929 10:24:52.590570 21638 mini_ranger_kms.cc:276] Ranger KMS PID: 329
I20260929 10:24:52.590732 21638 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.33.148:43789
Sep 29, 2026 10:24:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:24:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:24:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:24:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:24:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:24:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 10:24:54 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677494, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.33.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:54.820645 21638 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.224s user 0.019s sys 0.143s
I20260929 10:24:54.820776 21638 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 10:24:54.820879 21638 mini_ranger_kms.cc:401] 127.21.33.148:43789/kms/v1/keys
Sep 29 10:24:55 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677494, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.33.148@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
Sep 29 10:24:55 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677494, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.33.148@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
Sep 29 10:24:57 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:24:57.944218 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33481
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:33481
--ranger_config_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:43789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:24:58.052415 380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:58.052657 380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:58.052695 380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:58.056627 380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:24:58.056681 380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:58.056700 380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:58.056716 380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:24:58.056730 380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:24:58.062016 380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.33.148:43789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:33481
--ranger_config_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:58.063040 380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:58.064050 380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:58.073189 388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:58.073318 380 server_base.cc:1034] running on GCE node
W20260929 10:24:58.073190 385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:58.073282 386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:58.074118 380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:58.075155 380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:58.076318 380 hybrid_clock.cc:648] HybridClock initialized: now 1790677498076300 us; error 31 us; skew 500 ppm
Sep 29 10:24:58 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:58.079025 380 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:24:58.079973 380 webserver.cc:460] Webserver started at http://127.21.33.190:36309/ using document root <none> and password file <none>
I20260929 10:24:58.080503 380 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:58.080581 380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:58.080806 380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:24:58 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677498, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:24:58.135299 380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "98bbea04e52149f18196fd182b7746c8"
format_stamp: "Formatted at 2026-09-29 10:24:58 on dist-test-slave-2rbl"
server_key: "5649e0444d6f0594600e36e254be722a"
server_key_iv: "d779cb57abd766ba7744115c78c3688d"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:58.136204 380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "98bbea04e52149f18196fd182b7746c8"
format_stamp: "Formatted at 2026-09-29 10:24:58 on dist-test-slave-2rbl"
server_key: "5649e0444d6f0594600e36e254be722a"
server_key_iv: "d779cb57abd766ba7744115c78c3688d"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:58.141433 380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:24:58.159634 395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:58.161165 380 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:24:58.161372 380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "98bbea04e52149f18196fd182b7746c8"
format_stamp: "Formatted at 2026-09-29 10:24:58 on dist-test-slave-2rbl"
server_key: "5649e0444d6f0594600e36e254be722a"
server_key_iv: "d779cb57abd766ba7744115c78c3688d"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:58.161507 380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:58.179292 380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:58.181836 380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:58.182040 380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:58.189451 380 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:33481
I20260929 10:24:58.189478 447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:33481 every 8 connection(s)
I20260929 10:24:58.190378 380 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:24:58.191838 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 380
I20260929 10:24:58.191972 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:24:58.192268 21638 external_mini_cluster.cc:1428] Setting key 7c63ca6e67452fbe4a241cc87e945800
I20260929 10:24:58.193356 448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:58.198686 448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8: Bootstrap starting.
Sep 29 10:24:58 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:24:58.201072 448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:58.201908 448 log.cc:824] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:58.203743 448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8: No bootstrap required, opened a new log
I20260929 10:24:58.206055 451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:58.193676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.21.33.190:33481)
0929 10:24:58.194081 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:58.194095 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:58.194128 (+ 33us) server_negotiation.cc:407] Connection header received
0929 10:24:58.194749 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:58.194765 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:58.195068 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:58.195394 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:58.196180 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:58.197136 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:58.198028 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:58.198501 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:58.198662 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:58.201336 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:58.201341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:58.201353 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:58.201378 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:58.203434 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:58.203868 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:58.203875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:58.203886 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:58.204004 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:58.204288 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:58.204293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:58.204295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:58.204658 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:58.204866 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:58.205106 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 10:24:58.205389 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
I20260929 10:24:58.206768 448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bbea04e52149f18196fd182b7746c8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33481 } }
I20260929 10:24:58.207021 448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:58.207106 448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bbea04e52149f18196fd182b7746c8, State: Initialized, Role: FOLLOWER
I20260929 10:24:58.207551 448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bbea04e52149f18196fd182b7746c8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33481 } }
I20260929 10:24:58.207695 448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:24:58.207754 448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:24:58.207855 448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:24:58.208792 448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bbea04e52149f18196fd182b7746c8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33481 } }
I20260929 10:24:58.209066 448 leader_election.cc:302] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98bbea04e52149f18196fd182b7746c8; no voters:
I20260929 10:24:58.209383 448 leader_election.cc:288] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:24:58.209470 453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:24:58.209740 453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [term 1 LEADER]: Becoming Leader. State: Replica: 98bbea04e52149f18196fd182b7746c8, State: Running, Role: LEADER
I20260929 10:24:58.210074 453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bbea04e52149f18196fd182b7746c8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33481 } }
I20260929 10:24:58.210526 448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:24:58.211685 455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98bbea04e52149f18196fd182b7746c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bbea04e52149f18196fd182b7746c8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33481 } } }
I20260929 10:24:58.211815 455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:24:58.212174 462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:24:58.212133 454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98bbea04e52149f18196fd182b7746c8. Latest consensus state: current_term: 1 leader_uuid: "98bbea04e52149f18196fd182b7746c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98bbea04e52149f18196fd182b7746c8" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33481 } } }
I20260929 10:24:58.212710 454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:24:58.215241 462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:24:58.215679 448 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:24:58.227123 462 catalog_manager.cc:1347] Generated new cluster ID: 04b50a8413384856a8a4f9018a0b172b
I20260929 10:24:58.227207 462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:24:58.255184 462 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:24:58.256472 462 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:24:58.270800 462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8: Generated new TSK 0
I20260929 10:24:58.271657 462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:24:58.907311 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:43789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33481
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:24:59.031847 493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:59.032152 493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:59.032231 493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:59.036665 493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:59.036767 493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:59.036875 493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:24:59.042943 493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.33.148:43789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:59.044015 493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:59.045089 493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:59.055977 499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:59.056094 498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:59.055963 501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:59.056252 493 server_base.cc:1034] running on GCE node
I20260929 10:24:59.056730 493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:59.057415 493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:59.058595 493 hybrid_clock.cc:648] HybridClock initialized: now 1790677499058566 us; error 44 us; skew 500 ppm
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:59.061978 493 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:24:59.063005 493 webserver.cc:460] Webserver started at http://127.21.33.129:43619/ using document root <none> and password file <none>
I20260929 10:24:59.063692 493 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:59.063786 493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:59.064079 493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:24:59.079653 493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "6e0fdbef14ae4990ba461ff5fdb745d6"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "65f4ee4fed5e22b365c5ad066ae37f79"
server_key_iv: "3740d9f6207826c0df53c838a0ee2180"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.080307 493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "6e0fdbef14ae4990ba461ff5fdb745d6"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "65f4ee4fed5e22b365c5ad066ae37f79"
server_key_iv: "3740d9f6207826c0df53c838a0ee2180"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.084857 493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 10:24:59.104079 510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.105754 493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:24:59.105963 493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "6e0fdbef14ae4990ba461ff5fdb745d6"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "65f4ee4fed5e22b365c5ad066ae37f79"
server_key_iv: "3740d9f6207826c0df53c838a0ee2180"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.106206 493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:59.123668 493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:59.127162 493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:59.127388 493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:59.128037 493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:59.128937 493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:59.129006 493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.129104 493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:59.129141 493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.140965 493 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:36229
I20260929 10:24:59.142278 493 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:24:59.144142 623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:36229 every 8 connection(s)
I20260929 10:24:59.150038 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 493
I20260929 10:24:59.150158 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:24:59.150496 21638 external_mini_cluster.cc:1428] Setting key 4fdec465c77408994fef872c40c95553
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:24:59.179170 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.148187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33481 (local address 127.21.33.129:32939)
0929 10:24:59.153518 (+ 5331us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:59.153586 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:59.154627 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:59.165555 (+ 10928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:59.165567 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:59.166077 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:59.166915 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.166941 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.168295 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.168299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.169784 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.169796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.170607 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.170613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.170823 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.171578 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:59.171604 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:59.173332 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:59.175732 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:59.175742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:59.175762 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:59.176135 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:59.176479 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:59.176485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:59.176489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:59.176687 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:59.177136 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:59.177143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:59.177506 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 10:24:59.177714 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 10:24:59.178069 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5069,"thread_start_us":170,"threads_started":1}
I20260929 10:24:59.179922 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.149370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:32939 (local address 127.21.33.190:33481)
0929 10:24:59.164750 (+ 15380us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:59.164760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:59.164780 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:24:59.164874 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:59.164879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:59.164954 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:59.165078 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:59.167131 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.168106 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.169927 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.170282 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.170391 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.173557 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:59.173562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:59.173567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:59.173597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:59.175593 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.176267 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.176274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.176278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.176375 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.176791 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.176794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.176795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.177004 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:59.177087 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:59.179617 (+ 2530us) server_negotiation.cc:299] Negotiation successful
0929 10:24:59.179750 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15264,"thread_start_us":119,"threads_started":1}
I20260929 10:24:59.181770 624 heartbeater.cc:347] Connected to a master server at 127.21.33.190:33481
I20260929 10:24:59.182024 624 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:59.182629 624 heartbeater.cc:511] Master 127.21.33.190:33481 requested a full tablet report, sending...
I20260929 10:24:59.184542 412 ts_manager.cc:194] Registered new tserver with Master: 6e0fdbef14ae4990ba461ff5fdb745d6 (127.21.33.129:36229)
I20260929 10:24:59.186043 412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:32939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:24:59.205472 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:43789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33481
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 10:24:59.372001 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:59.372396 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:59.372464 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:59.378861 632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:59.378952 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:59.379041 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:24:59.387668 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.33.148:43789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:59.389219 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:59.390658 632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:59.403215 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:59.403170 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:59.403157 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:59.403681 632 server_base.cc:1034] running on GCE node
I20260929 10:24:59.404198 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:59.405061 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:59.406271 632 hybrid_clock.cc:648] HybridClock initialized: now 1790677499406248 us; error 35 us; skew 500 ppm
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:59.410521 632 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:24:59.411733 632 webserver.cc:460] Webserver started at http://127.21.33.130:36075/ using document root <none> and password file <none>
I20260929 10:24:59.412570 632 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:59.412662 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:59.412961 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:24:59.431474 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "8119554b3f8b42029dfc2b7be2372a88"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "96cb1d7aaab36842998fd25e29b9b899"
server_key_iv: "dfd4a709bfa3788e8bc601b2bfd8d708"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.432284 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "8119554b3f8b42029dfc2b7be2372a88"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "96cb1d7aaab36842998fd25e29b9b899"
server_key_iv: "dfd4a709bfa3788e8bc601b2bfd8d708"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.437990 632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 10:24:59.457659 648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.459857 632 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260929 10:24:59.460003 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "8119554b3f8b42029dfc2b7be2372a88"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "96cb1d7aaab36842998fd25e29b9b899"
server_key_iv: "dfd4a709bfa3788e8bc601b2bfd8d708"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.460144 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:59.478940 632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:59.482877 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:59.483143 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:59.483940 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:59.485041 632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:59.485148 632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.485291 632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:59.485390 632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.498327 632 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:36473
I20260929 10:24:59.499609 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:24:59.499888 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 632
I20260929 10:24:59.500036 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:24:59.500349 21638 external_mini_cluster.cc:1428] Setting key bce1375080994268b3a5f874039392b3
I20260929 10:24:59.500708 761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:36473 every 8 connection(s)
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:24:59.536563 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.508889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:54153 (local address 127.21.33.190:33481)
0929 10:24:59.509042 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:59.509048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:59.511438 (+ 2390us) server_negotiation.cc:407] Connection header received
0929 10:24:59.514386 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:59.514390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:59.514448 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:59.514505 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:59.521320 (+ 6815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.522172 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.524127 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.524492 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.524576 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.529675 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:59.529679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:59.529682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:59.529714 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:59.531608 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.534501 (+ 2893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.534506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.534508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.534614 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.535123 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.535127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.535128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.535388 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:59.535475 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:59.536233 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 10:24:59.536386 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:24:59.538759 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.507725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33481 (local address 127.21.33.130:54153)
0929 10:24:59.511247 (+ 3522us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:59.511313 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:59.514080 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:59.514965 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:59.514992 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:59.515373 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:59.516055 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.516073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.522384 (+ 6311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.522388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.523959 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.523977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.524807 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.524811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.525034 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.525888 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:59.525909 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:59.529436 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:59.531777 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:59.531783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:59.531804 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:59.532194 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:59.534743 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:59.534746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:59.534749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:59.534990 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:59.535547 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:59.535554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:59.535914 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 10:24:59.537082 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0929 10:24:59.537504 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3255,"thread_start_us":173,"threads_started":1}
I20260929 10:24:59.540516 762 heartbeater.cc:347] Connected to a master server at 127.21.33.190:33481
I20260929 10:24:59.540719 762 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:59.541343 762 heartbeater.cc:511] Master 127.21.33.190:33481 requested a full tablet report, sending...
I20260929 10:24:59.542639 412 ts_manager.cc:194] Registered new tserver with Master: 8119554b3f8b42029dfc2b7be2372a88 (127.21.33.130:36473)
I20260929 10:24:59.543409 412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:54153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:24:59.569816 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:43789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33481
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:24:59.720875 767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:24:59.721395 767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:24:59.721540 767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:24:59.727948 767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:24:59.728091 767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:24:59.728243 767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:24:59.736428 767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.33.148:43789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:33481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:24:59.738221 767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:24:59.739710 767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:24:59.753427 773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:24:59.754374 772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:59.754786 767 server_base.cc:1034] running on GCE node
W20260929 10:24:59.755247 775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:24:59.755843 767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:24:59.756654 767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:24:59.758033 767 hybrid_clock.cc:648] HybridClock initialized: now 1790677499757948 us; error 99 us; skew 500 ppm
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:24:59.762193 767 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:24:59.763412 767 webserver.cc:460] Webserver started at http://127.21.33.131:43507/ using document root <none> and password file <none>
I20260929 10:24:59.764197 767 fs_manager.cc:359] Metadata directory not provided
I20260929 10:24:59.764343 767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:24:59.764647 767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:24:59.785614 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "bfa4153f9a0642bdad40e70c7429e9d6"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "b5b6a972a5dadd8cec18725ad5666bbb"
server_key_iv: "fbb34282489dd8b26627947f0de3d78c"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.786736 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "bfa4153f9a0642bdad40e70c7429e9d6"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "b5b6a972a5dadd8cec18725ad5666bbb"
server_key_iv: "fbb34282489dd8b26627947f0de3d78c"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.792438 767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 10:24:59.814013 782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.815816 767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:24:59.816020 767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "bfa4153f9a0642bdad40e70c7429e9d6"
format_stamp: "Formatted at 2026-09-29 10:24:59 on dist-test-slave-2rbl"
server_key: "b5b6a972a5dadd8cec18725ad5666bbb"
server_key_iv: "fbb34282489dd8b26627947f0de3d78c"
server_key_version: "kuduclusterkey@0"
I20260929 10:24:59.816251 767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:24:59.834164 767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:24:59.838092 767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:24:59.838361 767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:24:59.839182 767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:24:59.840319 767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:24:59.840428 767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.840535 767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:24:59.840610 767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:24:59.854920 767 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:46557
I20260929 10:24:59.856232 767 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:24:59.859644 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 767
I20260929 10:24:59.859800 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:24:59.860077 21638 external_mini_cluster.cc:1428] Setting key 9f9c83588ff0f7a6c6325870ff4c4191
I20260929 10:24:59.857808 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:46557 every 8 connection(s)
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:24:59.884073 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.860932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:48091 (local address 127.21.33.190:33481)
0929 10:24:59.861056 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:59.861060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:59.863661 (+ 2601us) server_negotiation.cc:407] Connection header received
0929 10:24:59.864915 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:59.864918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:59.864970 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:59.865022 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:59.867326 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.868154 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.869931 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.870296 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.870376 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.875636 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:59.875640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:59.875643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:59.875671 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:59.877650 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.882122 (+ 4472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.882127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.882129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.882251 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.882719 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.882723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.882725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.882965 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:59.883050 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:59.883771 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 10:24:59.883903 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:24:59.886025 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.861528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33481 (local address 127.21.33.131:48091)
0929 10:24:59.863504 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:59.863565 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:59.864670 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:59.865471 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:59.865482 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:59.866443 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:24:59.867145 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.867166 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.868352 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.868356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.869789 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.869800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.870584 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.870588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.870787 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.872162 (+ 1375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:24:59.872196 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:24:59.875445 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:24:59.879001 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:59.879016 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:59.879036 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:59.879440 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:59.882377 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:24:59.882380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:24:59.882382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:24:59.882600 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:24:59.883118 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:24:59.883126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:24:59.883479 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 10:24:59.884516 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0929 10:24:59.884887 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"thread_start_us":154,"threads_started":1}
I20260929 10:24:59.888320 896 heartbeater.cc:347] Connected to a master server at 127.21.33.190:33481
I20260929 10:24:59.888589 896 heartbeater.cc:464] Registering TS with master...
I20260929 10:24:59.889154 896 heartbeater.cc:511] Master 127.21.33.190:33481 requested a full tablet report, sending...
I20260929 10:24:59.890270 412 ts_manager.cc:194] Registered new tserver with Master: bfa4153f9a0642bdad40e70c7429e9d6 (127.21.33.131:46557)
I20260929 10:24:59.890977 412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:48091
Sep 29 10:24:59 dist-test-slave-2rbl krb5kdc[32404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677499, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:24:59.897984 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:24:59.912240 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.901586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.21.33.190:33481)
0929 10:24:59.901722 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:59.901727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:59.902417 (+ 690us) server_negotiation.cc:407] Connection header received
0929 10:24:59.902623 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:59.902627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:59.902698 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:59.902756 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:24:59.903831 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.904659 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.905947 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.906275 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.906355 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.908252 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:24:59.908258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:24:59.908263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:24:59.908292 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:24:59.910183 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.910717 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.910721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.910723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.910814 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:24:59.911184 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:24:59.911188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:24:59.911189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:24:59.911411 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:24:59.911539 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:59.911952 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 10:24:59.912070 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:24:59.916406 412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:24:59.920967 412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:24:59.951583 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.934989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:46557 (local address 127.0.0.1:60016)
0929 10:24:59.945375 (+ 10386us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:59.945392 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:59.945510 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:59.946202 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:59.946209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:59.946246 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:59.946429 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.946440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.947967 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.947970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.949974 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.949982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.951142 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.951145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.951225 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.951256 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 10:24:59.951316 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 10:24:59.951424 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10305,"mutex_wait_us":779,"spinlock_wait_cycles":4608,"thread_start_us":92,"threads_started":1}
I20260929 10:24:59.951654 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.934325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.21.33.130:36473)
0929 10:24:59.945434 (+ 11109us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:59.945450 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:59.945480 (+ 30us) server_negotiation.cc:407] Connection header received
0929 10:24:59.945567 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:59.945574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:59.945836 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:59.946015 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:59.946775 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.947865 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.949824 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.950674 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.950777 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.950945 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:59.951421 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 10:24:59.951522 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10958,"thread_start_us":140,"threads_started":1}
I20260929 10:24:59.951987 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.934194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:36473 (local address 127.0.0.1:43246)
0929 10:24:59.937894 (+ 3700us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:59.937954 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:59.938085 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:59.946202 (+ 8117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:59.946209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:59.946247 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:59.946652 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.946658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.947974 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.947977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.949459 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.949470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.951007 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.951010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.951131 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.951173 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 10:24:59.951284 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 10:24:59.951857 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3501,"mutex_wait_us":740,"spinlock_wait_cycles":6400,"thread_start_us":116,"threads_started":1}
I20260929 10:24:59.952252 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.935415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.21.33.131:46557)
0929 10:24:59.937598 (+ 2183us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:59.937611 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:59.945617 (+ 8006us) server_negotiation.cc:407] Connection header received
0929 10:24:59.945702 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:59.945706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:59.945864 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:59.946014 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:59.946575 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.947811 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.950115 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.950868 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.950968 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.951089 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:59.951977 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 10:24:59.952088 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":134,"threads_started":1}
I20260929 10:24:59.955176 696 tablet_service.cc:1467] Processing CreateTablet for tablet f4a9d3e460974dccb86b2285ea130071 (DEFAULT_TABLE table=test-table [id=a07e58c73d8545ec8b48ba4c41fb5c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:24:59.955178 830 tablet_service.cc:1467] Processing CreateTablet for tablet f4a9d3e460974dccb86b2285ea130071 (DEFAULT_TABLE table=test-table [id=a07e58c73d8545ec8b48ba4c41fb5c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:24:59.956311 830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a9d3e460974dccb86b2285ea130071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:59.956319 696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a9d3e460974dccb86b2285ea130071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:59.956745 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.936639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:36229 (local address 127.0.0.1:49770)
0929 10:24:59.946389 (+ 9750us) negotiation.cc:107] Waiting for socket to connect
0929 10:24:59.946403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:24:59.946485 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:24:59.949582 (+ 3097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:24:59.949585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:24:59.949603 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:24:59.949685 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.949690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.953683 (+ 3993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.953687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.955110 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:24:59.955122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.956354 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.956359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:24:59.956443 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.956467 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:24:59.956540 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:24:59.956609 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9679,"thread_start_us":96,"threads_started":1}
I20260929 10:24:59.957250 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:24:59.937112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.21.33.129:36229)
0929 10:24:59.948246 (+ 11134us) server_negotiation.cc:206] Beginning negotiation
0929 10:24:59.948253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:24:59.948277 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:24:59.948353 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:24:59.948359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:24:59.948581 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:24:59.948733 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:24:59.952460 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.953564 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.955265 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:24:59.956037 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:24:59.956137 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:24:59.956276 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 10:24:59.957004 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 10:24:59.957098 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11025,"thread_start_us":119,"threads_started":1}
I20260929 10:24:59.960028 558 tablet_service.cc:1467] Processing CreateTablet for tablet f4a9d3e460974dccb86b2285ea130071 (DEFAULT_TABLE table=test-table [id=a07e58c73d8545ec8b48ba4c41fb5c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:24:59.961117 558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a9d3e460974dccb86b2285ea130071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:24:59.979110 915 tablet_bootstrap.cc:492] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Bootstrap starting.
I20260929 10:24:59.979570 917 tablet_bootstrap.cc:492] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6: Bootstrap starting.
I20260929 10:24:59.984279 917 tablet_bootstrap.cc:654] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:59.984768 916 tablet_bootstrap.cc:492] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: Bootstrap starting.
I20260929 10:24:59.987509 916 tablet_bootstrap.cc:654] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:59.989096 917 log.cc:824] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:59.989399 915 tablet_bootstrap.cc:654] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Neither blocks nor log segments found. Creating new log.
I20260929 10:24:59.990309 915 log.cc:824] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:59.990310 916 log.cc:824] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: Log is configured to *not* fsync() on all Append() calls
I20260929 10:24:59.993042 917 tablet_bootstrap.cc:492] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6: No bootstrap required, opened a new log
I20260929 10:24:59.993319 917 ts_tablet_manager.cc:1397] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20260929 10:24:59.993924 915 tablet_bootstrap.cc:492] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: No bootstrap required, opened a new log
I20260929 10:24:59.994099 915 ts_tablet_manager.cc:1397] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20260929 10:24:59.997076 916 tablet_bootstrap.cc:492] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: No bootstrap required, opened a new log
I20260929 10:24:59.997295 916 ts_tablet_manager.cc:1397] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260929 10:24:59.998090 917 raft_consensus.cc:348] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:24:59.998402 917 raft_consensus.cc:374] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:59.998484 917 raft_consensus.cc:729] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0fdbef14ae4990ba461ff5fdb745d6, State: Initialized, Role: FOLLOWER
I20260929 10:24:59.998090 915 raft_consensus.cc:348] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:24:59.998667 915 raft_consensus.cc:374] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:24:59.998761 915 raft_consensus.cc:729] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa4153f9a0642bdad40e70c7429e9d6, State: Initialized, Role: FOLLOWER
I20260929 10:24:59.999027 917 consensus_queue.cc:260] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:24:59.999528 915 consensus_queue.cc:260] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:25:00.000216 917 ts_tablet_manager.cc:1428] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:25:00.000469 915 ts_tablet_manager.cc:1428] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 10:25:00.000631 624 heartbeater.cc:503] Master 127.21.33.190:33481 was elected leader, sending a full tablet report...
I20260929 10:25:00.000864 896 heartbeater.cc:503] Master 127.21.33.190:33481 was elected leader, sending a full tablet report...
I20260929 10:25:00.001478 916 raft_consensus.cc:348] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:25:00.001811 916 raft_consensus.cc:374] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:00.001880 916 raft_consensus.cc:729] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8119554b3f8b42029dfc2b7be2372a88, State: Initialized, Role: FOLLOWER
I20260929 10:25:00.002272 916 consensus_queue.cc:260] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:25:00.003274 916 ts_tablet_manager.cc:1428] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:25:00.003621 762 heartbeater.cc:503] Master 127.21.33.190:33481 was elected leader, sending a full tablet report...
I20260929 10:25:00.004352 921 raft_consensus.cc:482] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:25:00.004531 921 raft_consensus.cc:504] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:25:00.005749 921 leader_election.cc:288] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8119554b3f8b42029dfc2b7be2372a88 (127.21.33.130:36473), bfa4153f9a0642bdad40e70c7429e9d6 (127.21.33.131:46557)
I20260929 10:25:00.010671 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:00.006483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45445 (local address 127.21.33.131:46557)
0929 10:25:00.006596 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:00.006601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:00.006612 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:25:00.006656 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:00.006658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:00.006690 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:00.006769 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:00.007089 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.007800 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.009391 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.009932 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.010034 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:00.010098 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:00.010474 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 10:25:00.010550 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:25:00.011157 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:00.006118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:46557 (local address 127.21.33.129:45445)
0929 10:25:00.006392 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:00.006408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:00.006512 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:00.006793 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:00.006795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:00.006811 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:00.006961 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:00.006972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.007973 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.007976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:00.009241 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:00.009250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.010208 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.010212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:00.010320 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:00.010358 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 10:25:00.010885 (+ 527us) client_negotiation.cc:240] Negotiation successful
0929 10:25:00.011019 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":120,"threads_started":1}
W20260929 10:25:00.011454 763 tablet.cc:2369] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:25:00.012537 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a9d3e460974dccb86b2285ea130071" candidate_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" is_pre_election: true
I20260929 10:25:00.012850 834 raft_consensus.cc:2454] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e0fdbef14ae4990ba461ff5fdb745d6 in term 0.
I20260929 10:25:00.013505 512 leader_election.cc:302] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0fdbef14ae4990ba461ff5fdb745d6, bfa4153f9a0642bdad40e70c7429e9d6; no voters:
I20260929 10:25:00.013866 921 raft_consensus.cc:2781] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:25:00.013989 921 raft_consensus.cc:482] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:25:00.014079 921 raft_consensus.cc:3037] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:00.015450 921 raft_consensus.cc:504] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:25:00.015537 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:00.007361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:52911 (local address 127.21.33.130:36473)
0929 10:25:00.007461 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:00.007466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:00.011484 (+ 4018us) server_negotiation.cc:407] Connection header received
0929 10:25:00.011692 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:00.011698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:00.011753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:00.011801 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:00.012158 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.012831 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.014234 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.014803 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.014894 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:00.014945 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:00.015318 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 10:25:00.015418 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:25:00.015846 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:00.007050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:36473 (local address 127.21.33.129:52911)
0929 10:25:00.011411 (+ 4361us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:00.011427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:00.011575 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:00.011921 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:00.011924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:00.011938 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:00.012006 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:00.012010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.012987 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.012990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:00.014094 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:00.014104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.015035 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.015038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:00.015174 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:00.015196 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:25:00.015652 (+ 456us) client_negotiation.cc:240] Negotiation successful
0929 10:25:00.015733 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4260,"thread_start_us":82,"threads_started":1}
I20260929 10:25:00.016495 712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a9d3e460974dccb86b2285ea130071" candidate_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8119554b3f8b42029dfc2b7be2372a88"
I20260929 10:25:00.016444 711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a9d3e460974dccb86b2285ea130071" candidate_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8119554b3f8b42029dfc2b7be2372a88" is_pre_election: true
I20260929 10:25:00.016672 712 raft_consensus.cc:3037] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:00.017493 921 leader_election.cc:288] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [CANDIDATE]: Term 1 election: Requested vote from peers 8119554b3f8b42029dfc2b7be2372a88 (127.21.33.130:36473), bfa4153f9a0642bdad40e70c7429e9d6 (127.21.33.131:46557)
I20260929 10:25:00.017985 712 raft_consensus.cc:2454] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e0fdbef14ae4990ba461ff5fdb745d6 in term 1.
I20260929 10:25:00.018764 511 leader_election.cc:302] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0fdbef14ae4990ba461ff5fdb745d6, 8119554b3f8b42029dfc2b7be2372a88; no voters:
I20260929 10:25:00.019026 834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a9d3e460974dccb86b2285ea130071" candidate_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa4153f9a0642bdad40e70c7429e9d6"
I20260929 10:25:00.019164 834 raft_consensus.cc:3037] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:00.021019 834 raft_consensus.cc:2454] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e0fdbef14ae4990ba461ff5fdb745d6 in term 1.
I20260929 10:25:00.023401 924 raft_consensus.cc:2781] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:00.023671 924 raft_consensus.cc:686] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [term 1 LEADER]: Becoming Leader. State: Replica: 6e0fdbef14ae4990ba461ff5fdb745d6, State: Running, Role: LEADER
I20260929 10:25:00.024377 924 consensus_queue.cc:237] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } }
I20260929 10:25:00.027673 411 catalog_manager.cc:5495] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e0fdbef14ae4990ba461ff5fdb745d6 (127.21.33.129). New cstate: current_term: 1 leader_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e0fdbef14ae4990ba461ff5fdb745d6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 36229 } health_report { overall_health: HEALTHY } } }
I20260929 10:25:00.058126 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:00.053410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.21.33.129:36229)
0929 10:25:00.053528 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:00.053533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:00.053622 (+ 89us) server_negotiation.cc:407] Connection header received
0929 10:25:00.053799 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:00.053803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:00.053876 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:00.053994 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:25:00.054288 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.055077 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.056292 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:00.056635 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:00.056742 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:00.057092 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:25:00.057700 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:25:00.057828 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:00.057908 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 10:25:00.057974 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:25:00.065349 712 raft_consensus.cc:1261] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 1 FOLLOWER]: Refusing update from remote peer 6e0fdbef14ae4990ba461ff5fdb745d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:00.065389 834 raft_consensus.cc:1261] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 1 FOLLOWER]: Refusing update from remote peer 6e0fdbef14ae4990ba461ff5fdb745d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:00.066007 924 consensus_queue.cc:1035] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa4153f9a0642bdad40e70c7429e9d6" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:00.066195 921 consensus_queue.cc:1035] T f4a9d3e460974dccb86b2285ea130071 P 6e0fdbef14ae4990ba461ff5fdb745d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8119554b3f8b42029dfc2b7be2372a88" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:00.082276 930 mvcc.cc:205] Tried to move back new op lower bound from 7334615040259637248 to 7334615040102871040. Current Snapshot: MvccSnapshot[applied={T|T < 7334615040259637248}]
I20260929 10:25:00.083554 932 mvcc.cc:205] Tried to move back new op lower bound from 7334615040259637248 to 7334615040102871040. Current Snapshot: MvccSnapshot[applied={T|T < 7334615040259637248}]
I20260929 10:25:00.083772 929 mvcc.cc:205] Tried to move back new op lower bound from 7334615040259637248 to 7334615040102871040. Current Snapshot: MvccSnapshot[applied={T|T < 7334615040259637248}]
I20260929 10:25:00.095000 411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:00.095274 411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:00.097970 411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98bbea04e52149f18196fd182b7746c8: Sending DeleteTablet for 3 replicas of tablet f4a9d3e460974dccb86b2285ea130071
I20260929 10:25:00.098788 696 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a9d3e460974dccb86b2285ea130071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:00 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:43246
I20260929 10:25:00.099761 830 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a9d3e460974dccb86b2285ea130071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:00 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:60016
I20260929 10:25:00.100607 558 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a9d3e460974dccb86b2285ea130071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:00 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:49770
I20260929 10:25:00.102124 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 493
I20260929 10:25:00.104188 940 tablet_replica.cc:330] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: stopping tablet replica
I20260929 10:25:00.104429 939 tablet_replica.cc:330] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: stopping tablet replica
I20260929 10:25:00.104851 940 raft_consensus.cc:2229] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:00.105232 939 raft_consensus.cc:2229] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:00.105600 939 raft_consensus.cc:2258] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:00.105955 940 raft_consensus.cc:2258] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:00.107816 940 ts_tablet_manager.cc:1905] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:00.109112 939 ts_tablet_manager.cc:1905] T f4a9d3e460974dccb86b2285ea130071 P 8119554b3f8b42029dfc2b7be2372a88: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:25:00.111451 396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:36229 (error 108)
I20260929 10:25:00.112876 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 632
I20260929 10:25:00.112949 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:00.112626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:36229 (local address 127.0.0.1:49776)
0929 10:25:00.112755 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:00.112839 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 10:25:00.114606 396 catalog_manager.cc:4584] TS 6e0fdbef14ae4990ba461ff5fdb745d6 (127.21.33.129:36229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4a9d3e460974dccb86b2285ea130071: Network error: Client connection negotiation failed: client connection to 127.21.33.129:36229: connect: Connection refused (error 111)
I20260929 10:25:00.114822 940 ts_tablet_manager.cc:1918] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:00.114949 940 log.cc:1196] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/f4a9d3e460974dccb86b2285ea130071
I20260929 10:25:00.115290 940 ts_tablet_manager.cc:1939] T f4a9d3e460974dccb86b2285ea130071 P bfa4153f9a0642bdad40e70c7429e9d6: Deleting consensus metadata
I20260929 10:25:00.116707 397 catalog_manager.cc:4841] TS bfa4153f9a0642bdad40e70c7429e9d6 (127.21.33.131:46557): tablet f4a9d3e460974dccb86b2285ea130071 (table test-table [id=a07e58c73d8545ec8b48ba4c41fb5c98]) successfully deleted
I20260929 10:25:00.120095 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:00.119799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:36473 (local address 127.0.0.1:43248)
0929 10:25:00.119945 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:00.120006 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.130:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 10:25:00.120245 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 767
I20260929 10:25:00.127466 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 380
I20260929 10:25:00.133003 21638 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 10:25:00.478423 21638 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 10:25:00.478546 21638 mini_ranger.cc:66] Stopping Ranger...
I20260929 10:25:00.823520 21638 mini_ranger.cc:68] Stopped Ranger
2026-09-29 10:25:00.823 UTC [32434] LOG: received smart shutdown request
2026-09-29 10:25:00.827 UTC [32434] LOG: background worker "logical replication launcher" (PID 32443) exited with exit code 1
2026-09-29 10:25:00.827 UTC [32438] LOG: shutting down
2026-09-29 10:25:01.010 UTC [32434] LOG: database system is shut down
2026-09-29T10:25:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45465 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:25:01 dist-test-slave-2rbl krb5kdc[951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:25:01 dist-test-slave-2rbl krb5kdc[951](info): set up 2 sockets
Sep 29 10:25:01 dist-test-slave-2rbl krb5kdc[951](info): commencing operation
krb5kdc: starting...
W20260929 10:25:02.190014 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:25:02 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:25:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/rangeradmin_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/rangeradmin_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/rangerlookup_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/rangerlookup_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
I20260929 10:25:02.365100 21638 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksWWfjY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/postgres -l logfile start
2026-09-29 10:25:04.969 UTC [981] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 10:25:04.969 UTC [981] LOG: listening on IPv4 address "127.21.33.148", port 57107
2026-09-29 10:25:04.973 UTC [981] LOG: listening on Unix socket "/tmp/.s.PGSQL.57107"
2026-09-29 10:25:05.008 UTC [984] LOG: database system was shut down at 2026-09-29 10:25:03 UTC
2026-09-29 10:25:05.013 UTC [981] LOG: database system is ready to accept connections
I20260929 10:25:06.009634 21638 mini_postgres.cc:96] Postgres bound to 57107
2026-09-29 10:25:06.015 UTC [992] FATAL: database "slave" does not exist
127.21.33.148:57107 - accepting connections
I20260929 10:25:06.016392 21638 mini_ranger.cc:161] Starting Ranger...
I20260929 10:25:06.031890 21638 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 10:25:06.542697 21638 mini_ranger.cc:87] Created ranger Postgres database
I20260929 10:25:06.542788 21638 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-admin
2026-09-29 10:25:06,816 [I] DB FLAVOR :POSTGRES
2026-09-29 10:25:06,816 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:25:06,816 [I] Checking connection..
2026-09-29 10:25:06,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 10:25:07,070 [I] Checking connection passed.
2026-09-29 10:25:07,070 [I] --------- Verifying version history table ---------
2026-09-29 10:25:07,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:25:07,333 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:25:07,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:25:07,579 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 10:25:07,580 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 10:25:07,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 10:25:07.804 UTC [1056] WARNING: there is no transaction in progress
2026-09-29 10:25:07,811 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 10:25:07,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 10:25:08,046 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 10:25:08,046 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:25:08,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 10:25:08,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 10:25:08,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2rbl.c.gcp-upstream.internal','N') ;"
2026-09-29 10:25:08,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:25:09,013 [I] Table x_portal_user does not exist in database ranger
2026-09-29 10:25:09,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:25:09,243 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 10:25:09,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:25:09,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:25:09,719 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 10:25:09,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 10:25:10.662 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.679 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.695 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.710 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.726 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.744 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.800 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.807 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.822 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.832 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.842 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.858 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.868 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.875 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.882 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:10.890 UTC [1164] WARNING: there is no transaction in progress
2026-09-29 10:25:11,584 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 10:25:11,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 10:25:11,807 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 10:25:11,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 10:25:12,067 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 10:25:12,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 10:25:12,298 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 10:25:12,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 10:25:12,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 10:25:12,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2rbl.c.gcp-upstream.internal';"
2026-09-29 10:25:13,025 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 10:25:13,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 10:25:13,272 [I] DB_PATCHES have already been applied
I20260929 10:25:13.278137 21638 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksWWfjY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 10:25:13.278218 21638 mini_ranger.cc:193] Using host: 127.21.33.148
I20260929 10:25:13.281777 21638 mini_ranger.cc:227] Ranger admin URL: http://127.21.33.148:49553
Sep 29, 2026 10:25:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:25:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:25:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:25:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:25:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:25:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:25:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 10:25:14.380143 21638 mini_ranger.cc:160] Time spent starting Ranger: real 8.364s user 0.002s sys 0.007s
Sep 29, 2026 10:25:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:25:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 10:25:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:25:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:25:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 10:25:35.545717 21638 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.33.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.33.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/rangerkms_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.33.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/rangerkms_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.33.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.33.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.33.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/HTTP_127.21.33.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 10:25:35 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 10:25:35.633282 21638 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 10:25:35.643591 21638 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 10:25:36.607029 21638 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 10:25:36.607136 21638 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms
I20260929 10:25:36.607159 21638 mini_ranger_kms.cc:227] Using postgres at 127.21.33.148:57107
2026-09-29 10:25:36,783 [I] DB FLAVOR :POSTGRES
2026-09-29 10:25:36,783 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 10:25:36,783 [I] Checking connection
2026-09-29 10:25:36,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 10:25:37,037 [I] connection success
2026-09-29 10:25:37,037 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 10:25:37,038 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 10:25:37,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 10:25:37,276 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 10:25:37,277 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 10:25:37,277 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 10:25:37,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.33.148:57107/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 10:25:37,557 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 10:25:38.044499 21638 mini_ranger_kms.cc:310] Created kms service
I20260929 10:25:38.230767 21638 mini_ranger_kms.cc:326] Created kudu user
I20260929 10:25:38.278270 21638 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 10:25:38 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:38.677838 21638 mini_ranger_kms.cc:384] Added ranger policy
I20260929 10:25:38.677997 21638 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksWWfjY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksWWfjY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 10:25:38.678018 21638 mini_ranger_kms.cc:240] Using host: 127.21.33.148
I20260929 10:25:38.682261 21638 mini_ranger_kms.cc:276] Ranger KMS PID: 1344
I20260929 10:25:38.682355 21638 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.33.148:36227
Sep 29, 2026 10:25:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:25:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:25:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:25:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:25:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:25:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 10:25:40.929675 21638 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.296s user 0.017s sys 0.142s
I20260929 10:25:40.929811 21638 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 10:25:40.929889 21638 mini_ranger_kms.cc:401] 127.21.33.148:36227/kms/v1/keys
Sep 29 10:25:41 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677541, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.33.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 10:25:41 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677541, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.33.148@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
Sep 29 10:25:41 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677541, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.33.148@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
Sep 29 10:25:44 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:25:44.220331 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46533
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:46533
--ranger_config_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:44.334435 1395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:44.334723 1395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:44.334764 1395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:44.338920 1395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:44.339025 1395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:44.339046 1395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:44.339061 1395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:44.339077 1395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:44.344827 1395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.33.148:36227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:46533
--ranger_config_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:44.346009 1395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:44.347093 1395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:25:44.356403 1395 server_base.cc:1034] running on GCE node
W20260929 10:25:44.356297 1401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:44.356353 1403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:44.356664 1400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:44.357339 1395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:44.358487 1395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:44.359669 1395 hybrid_clock.cc:648] HybridClock initialized: now 1790677544359651 us; error 32 us; skew 500 ppm
Sep 29 10:25:44 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:44.368606 1395 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:25:44.369680 1395 webserver.cc:460] Webserver started at http://127.21.33.190:46657/ using document root <none> and password file <none>
I20260929 10:25:44.370223 1395 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:44.370276 1395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:44.370447 1395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:25:44 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:44.428174 1395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "7ba62ee211a94281adbcf3ca3052979d"
format_stamp: "Formatted at 2026-09-29 10:25:44 on dist-test-slave-2rbl"
server_key: "03933d98b873216682bda9c276eaaf79"
server_key_iv: "8a574b412f52e6d90ff1c28eeb197111"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:44.428799 1395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "7ba62ee211a94281adbcf3ca3052979d"
format_stamp: "Formatted at 2026-09-29 10:25:44 on dist-test-slave-2rbl"
server_key: "03933d98b873216682bda9c276eaaf79"
server_key_iv: "8a574b412f52e6d90ff1c28eeb197111"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:44.432890 1395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 10:25:44.448027 1410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:44.449602 1395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 10:25:44.449761 1395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "7ba62ee211a94281adbcf3ca3052979d"
format_stamp: "Formatted at 2026-09-29 10:25:44 on dist-test-slave-2rbl"
server_key: "03933d98b873216682bda9c276eaaf79"
server_key_iv: "8a574b412f52e6d90ff1c28eeb197111"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:44.449913 1395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:44.476171 1395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:44.479239 1395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:44.479453 1395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:44.487780 1395 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:46533
I20260929 10:25:44.487785 1462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:46533 every 8 connection(s)
I20260929 10:25:44.488809 1395 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:25:44.491654 1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:44.497112 1463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Bootstrap starting.
I20260929 10:25:44.498380 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 1395
I20260929 10:25:44.498507 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:25:44.498799 21638 external_mini_cluster.cc:1428] Setting key 29b917b292590b4ca89783e85cc08553
I20260929 10:25:44.499902 1463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:44.500855 1463 log.cc:824] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:44.503044 1463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: No bootstrap required, opened a new log
I20260929 10:25:44.507227 1463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:44.507449 1463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:44.507483 1463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba62ee211a94281adbcf3ca3052979d, State: Initialized, Role: FOLLOWER
I20260929 10:25:44.507987 1463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:44.508114 1463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:25:44.508199 1463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:25:44.508301 1463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 0 FOLLOWER]: Advancing to term 1
Sep 29 10:25:44 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:44.509347 1463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:44.509745 1463 leader_election.cc:302] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ba62ee211a94281adbcf3ca3052979d; no voters:
I20260929 10:25:44.510160 1463 leader_election.cc:288] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:25:44.510260 1468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:44.510805 1468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 LEADER]: Becoming Leader. State: Replica: 7ba62ee211a94281adbcf3ca3052979d, State: Running, Role: LEADER
I20260929 10:25:44.511121 1468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:44.511548 1463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:44.512306 1470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ba62ee211a94281adbcf3ca3052979d. Latest consensus state: current_term: 1 leader_uuid: "7ba62ee211a94281adbcf3ca3052979d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } } }
I20260929 10:25:44.513087 1470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:44.512970 1469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ba62ee211a94281adbcf3ca3052979d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } } }
I20260929 10:25:44.513459 1469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:44.513825 1463 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:25:44.514503 1480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:25:44.517154 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:44.500002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.21.33.190:46533)
0929 10:25:44.500523 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:44.500541 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:44.500594 (+ 53us) server_negotiation.cc:407] Connection header received
0929 10:25:44.501445 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:44.501464 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:44.501860 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:44.502211 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:44.503069 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:44.504571 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:44.505814 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:44.506542 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:44.506719 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:44.509481 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:44.509487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:44.509505 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:44.509534 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:44.511527 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:44.512048 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:44.512053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:44.512060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:44.512189 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:44.512532 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:44.512536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:44.512537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:44.513120 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:44.513976 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:44.514131 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 10:25:44.514419 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":135,"threads_started":1}
I20260929 10:25:44.521628 1480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:25:44.529340 1480 catalog_manager.cc:1347] Generated new cluster ID: 48b1e06849e94f0fbb295541279dfa13
I20260929 10:25:44.529426 1480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:25:44.544721 1480 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:25:44.546443 1480 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:25:44.561182 1480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Generated new TSK 0
I20260929 10:25:44.561970 1480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:25:45.236560 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46533
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:45.396160 1508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:45.396615 1508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:45.396701 1508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:45.403338 1508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:45.403442 1508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:45.403546 1508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:25:45.412065 1508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.33.148:36227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:45.413635 1508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:45.415071 1508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:45.428076 1516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:45.428059 1518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:45.428079 1515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:45.428686 1508 server_base.cc:1034] running on GCE node
I20260929 10:25:45.429159 1508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:45.429888 1508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:45.431066 1508 hybrid_clock.cc:648] HybridClock initialized: now 1790677545431031 us; error 54 us; skew 500 ppm
Sep 29 10:25:45 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:45.434170 1508 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:25:45.435141 1508 webserver.cc:460] Webserver started at http://127.21.33.129:42847/ using document root <none> and password file <none>
I20260929 10:25:45.435719 1508 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:45.435781 1508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:45.435993 1508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:25:45 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:45.453910 1508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "68cd43362cba488f830e9a1dd49a6d2f"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "e53bb627114675bf9db7fdfd8429ff32"
server_key_iv: "f93f55e41b7c49414b1496cf83b4c8d0"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:45.454661 1508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "68cd43362cba488f830e9a1dd49a6d2f"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "e53bb627114675bf9db7fdfd8429ff32"
server_key_iv: "f93f55e41b7c49414b1496cf83b4c8d0"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:45.459605 1508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 10:25:45.485606 1525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:45.487363 1508 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:25:45.487497 1508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "68cd43362cba488f830e9a1dd49a6d2f"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "e53bb627114675bf9db7fdfd8429ff32"
server_key_iv: "f93f55e41b7c49414b1496cf83b4c8d0"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:45.487628 1508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:45.505630 1508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:45.509418 1508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:45.509625 1508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:45.510345 1508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:45.511418 1508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:45.511507 1508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:45.511600 1508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:45.511636 1508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:45.523701 1508 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:42401
I20260929 10:25:45.524981 1508 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:25:45.528694 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 1508
I20260929 10:25:45.528818 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:25:45.529109 21638 external_mini_cluster.cc:1428] Setting key cf119c0d3b6c5f95b79dd7d7ae03d518
I20260929 10:25:45.538260 1640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:42401 every 8 connection(s)
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Sep 29 10:25:45 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:25:45.567340 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:45.540842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46533 (local address 127.21.33.129:48827)
0929 10:25:45.546421 (+ 5579us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:45.546471 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:45.547589 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:45.548510 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:45.548535 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:45.548915 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:45.549763 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:45.549787 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.550988 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.550992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:45.552453 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:45.552465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.553301 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.553305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:45.553512 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:45.554312 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:45.554335 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:45.560769 (+ 6434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:45.563650 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:45.563657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:45.563677 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:45.564084 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:45.564424 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:45.564428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:45.564430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:45.564635 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:45.565153 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:45.565162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:45.565594 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 10:25:45.565808 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 10:25:45.566173 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5356,"thread_start_us":177,"threads_started":1}
I20260929 10:25:45.568387 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:45.542075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:48827 (local address 127.21.33.190:46533)
0929 10:25:45.547046 (+ 4971us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:45.547056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:45.547075 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:25:45.547980 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:45.547986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:45.548056 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:45.548156 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:45.549963 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.550822 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.552621 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.552984 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.553093 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:45.561041 (+ 7948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:45.561046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:45.561049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:45.561087 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:45.563451 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:45.564211 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:45.564215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:45.564217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:45.564320 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:45.564752 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:45.564755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:45.564757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:45.564995 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:45.565090 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:45.568052 (+ 2962us) server_negotiation.cc:299] Negotiation successful
0929 10:25:45.568189 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4874,"thread_start_us":111,"threads_started":1}
I20260929 10:25:45.570302 1641 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46533
I20260929 10:25:45.570561 1641 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:45.571149 1641 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:45.572909 1427 ts_manager.cc:194] Registered new tserver with Master: 68cd43362cba488f830e9a1dd49a6d2f (127.21.33.129:42401)
I20260929 10:25:45.574502 1427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:48827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:25:45.592730 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46533
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:25:45 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 10:25:45.758011 1647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:45.758482 1647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:45.758612 1647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:45.765064 1647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:45.765257 1647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:45.765405 1647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:25:45.773723 1647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.33.148:36227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:45.775434 1647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:45.776921 1647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:45.790459 1653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:45.792376 1656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:45.793808 1654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:45.795430 1647 server_base.cc:1034] running on GCE node
I20260929 10:25:45.795972 1647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:45.796855 1647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:45.798069 1647 hybrid_clock.cc:648] HybridClock initialized: now 1790677545798045 us; error 37 us; skew 500 ppm
Sep 29 10:25:45 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:45.801836 1647 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:25:45.802946 1647 webserver.cc:460] Webserver started at http://127.21.33.130:43299/ using document root <none> and password file <none>
I20260929 10:25:45.803694 1647 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:45.803809 1647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:45.804095 1647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:25:45 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:45.824757 1647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "682ced8e05c94f5b845d44c54342f781"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "b3ae4679c9854a3983592b9f5b631f8d"
server_key_iv: "ae10a5ff5db57d9feb290f47b8ab1944"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:45.826408 1647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "682ced8e05c94f5b845d44c54342f781"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "b3ae4679c9854a3983592b9f5b631f8d"
server_key_iv: "ae10a5ff5db57d9feb290f47b8ab1944"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:45.831590 1647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 10:25:45.857754 1663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:45.859756 1647 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260929 10:25:45.859916 1647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "682ced8e05c94f5b845d44c54342f781"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "b3ae4679c9854a3983592b9f5b631f8d"
server_key_iv: "ae10a5ff5db57d9feb290f47b8ab1944"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:45.860086 1647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:45.876878 1647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:45.881045 1647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:45.881356 1647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:45.882195 1647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:45.883328 1647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:45.883452 1647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:45.883561 1647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:45.883634 1647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:45.895172 1647 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:42637
I20260929 10:25:45.896390 1647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:25:45.902086 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 1647
I20260929 10:25:45.902207 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:25:45.902516 21638 external_mini_cluster.cc:1428] Setting key 99846c53e3af6013a97301b5714935a7
I20260929 10:25:45.907189 1776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:42637 every 8 connection(s)
Sep 29 10:25:45 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:45.925392 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:45.905394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:51009 (local address 127.21.33.190:46533)
0929 10:25:45.905530 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:45.905536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:45.906844 (+ 1308us) server_negotiation.cc:407] Connection header received
0929 10:25:45.908171 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:45.908175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:45.908231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:45.908285 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:45.912070 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.912746 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.914125 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.914374 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.914487 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:45.917025 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:45.917030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:45.917033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:45.917086 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:45.918803 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:45.921227 (+ 2424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:45.921233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:45.921237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:45.921331 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:45.921812 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:45.921818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:45.921820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:45.922110 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:45.922205 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:45.925014 (+ 2809us) server_negotiation.cc:299] Negotiation successful
0929 10:25:45.925183 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:25:45.927184 1780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:45.906073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46533 (local address 127.21.33.130:51009)
0929 10:25:45.906680 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:45.906744 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:45.907875 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:45.908709 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:45.908718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:45.911277 (+ 2559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:45.911899 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:45.911915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.912884 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.912889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:45.914000 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:45.914010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:45.914540 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:45.914544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:45.914713 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:45.915376 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:45.915392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:45.916807 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:45.920605 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:45.920615 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:45.920635 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:45.921064 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:45.921459 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:45.921463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:45.921465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:45.921685 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:45.924009 (+ 2324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:45.924019 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:45.924655 (+ 636us) client_negotiation.cc:769] Sending connection context
0929 10:25:45.925656 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0929 10:25:45.926048 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":166,"threads_started":1}
I20260929 10:25:45.929142 1777 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46533
I20260929 10:25:45.929442 1777 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:45.929921 1777 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:45.931005 1427 ts_manager.cc:194] Registered new tserver with Master: 682ced8e05c94f5b845d44c54342f781 (127.21.33.130:42637)
I20260929 10:25:45.931602 1427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:51009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:25:45.954439 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46533
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:46.124536 1782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:46.125025 1782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:46.125171 1782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:46.130511 1782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:46.130615 1782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:46.130728 1782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:25:46.138108 1782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.33.148:36227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:46.139989 1782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:46.141515 1782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:46.159494 1788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:46.159920 1790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:46.160548 1782 server_base.cc:1034] running on GCE node
W20260929 10:25:46.160271 1787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:46.161231 1782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:46.162076 1782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:46.163370 1782 hybrid_clock.cc:648] HybridClock initialized: now 1790677546163304 us; error 84 us; skew 500 ppm
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:46.167279 1782 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:25:46.168399 1782 webserver.cc:460] Webserver started at http://127.21.33.131:35911/ using document root <none> and password file <none>
I20260929 10:25:46.168979 1782 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:46.169051 1782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:46.169346 1782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:46.208673 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "1c40180790274e8aa6e88fd2ebe7286f"
format_stamp: "Formatted at 2026-09-29 10:25:46 on dist-test-slave-2rbl"
server_key: "7c0d5ff1790e24c7bc0fa949c8fd03b5"
server_key_iv: "b371c3f7f2c5ea4d1c444cda773fc64b"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:46.209576 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "1c40180790274e8aa6e88fd2ebe7286f"
format_stamp: "Formatted at 2026-09-29 10:25:46 on dist-test-slave-2rbl"
server_key: "7c0d5ff1790e24c7bc0fa949c8fd03b5"
server_key_iv: "b371c3f7f2c5ea4d1c444cda773fc64b"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:46.215204 1782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 10:25:46.233558 1797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:46.235201 1782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:25:46.235332 1782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "1c40180790274e8aa6e88fd2ebe7286f"
format_stamp: "Formatted at 2026-09-29 10:25:46 on dist-test-slave-2rbl"
server_key: "7c0d5ff1790e24c7bc0fa949c8fd03b5"
server_key_iv: "b371c3f7f2c5ea4d1c444cda773fc64b"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:46.235486 1782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:46.251344 1782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:46.254966 1782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:46.255163 1782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:46.255779 1782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:46.256752 1782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:46.256811 1782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:46.256866 1782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:46.256896 1782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:46.267664 1782 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:46215
I20260929 10:25:46.268774 1782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:25:46.275058 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 1782
I20260929 10:25:46.275197 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:25:46.275569 21638 external_mini_cluster.cc:1428] Setting key 562775db53240eed96258363e2d7299f
I20260929 10:25:46.276980 1910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:46215 every 8 connection(s)
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:46.291128 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:46.277911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:39425 (local address 127.21.33.190:46533)
0929 10:25:46.278112 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:46.278117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:46.278793 (+ 676us) server_negotiation.cc:407] Connection header received
0929 10:25:46.279934 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:46.279938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:46.280003 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:46.280118 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:46.281449 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.282231 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.283864 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.284191 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.284300 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:46.286988 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:46.286993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:46.286996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:46.287025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:46.288881 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:46.289555 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:46.289560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:46.289562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:46.289647 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:46.290017 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:46.290020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:46.290021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:46.290226 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:46.290322 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:46.290820 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 10:25:46.290963 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 10:25:46.291903 1913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:46.278044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46533 (local address 127.21.33.131:39425)
0929 10:25:46.278625 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:46.278700 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:46.279654 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:46.280302 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:46.280309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:46.280601 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:46.281283 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:46.281306 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.282380 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.282384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:46.283727 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:46.283739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.284363 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.284368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:46.284576 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:46.285318 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:46.285342 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:46.286824 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:46.289002 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:46.289008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:46.289029 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:46.289463 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:46.289741 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:46.289745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:46.289747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:46.289888 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:46.290318 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:46.290324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:46.290579 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 10:25:46.290765 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 10:25:46.291078 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":170,"threads_started":1}
I20260929 10:25:46.293329 1911 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46533
I20260929 10:25:46.293596 1911 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:46.294143 1911 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:46.295270 1427 ts_manager.cc:194] Registered new tserver with Master: 1c40180790274e8aa6e88fd2ebe7286f (127.21.33.131:46215)
I20260929 10:25:46.295845 1427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:39425
I20260929 10:25:46.302145 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:25:46.302775 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 1508
I20260929 10:25:46.308099 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 1647
I20260929 10:25:46.313966 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 1782
I20260929 10:25:46.320842 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 1395
I20260929 10:25:46.327211 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46533
--webserver_interface=127.21.33.190
--webserver_port=46657
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:46533
--ranger_config_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:46.442214 1914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:46.442649 1914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:46.442736 1914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:46.447154 1914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:46.447214 1914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:46.447234 1914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:46.447250 1914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:46.447295 1914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:46.452910 1914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:46533
--ranger_config_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=46657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:46.454013 1914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:46.455021 1914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:25:46.463663 1914 server_base.cc:1034] running on GCE node
W20260929 10:25:46.463672 1920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:46.463593 1922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:46.463634 1919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:46.464485 1914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:46.465679 1914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:46.466878 1914 hybrid_clock.cc:648] HybridClock initialized: now 1790677546466843 us; error 46 us; skew 500 ppm
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:46.469694 1914 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:25:46.470580 1914 webserver.cc:460] Webserver started at http://127.21.33.190:46657/ using document root <none> and password file <none>
I20260929 10:25:46.471124 1914 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:46.471217 1914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:46.534833 1914 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:25:46.537147 1930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:46.538436 1914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:46.538580 1914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "7ba62ee211a94281adbcf3ca3052979d"
format_stamp: "Formatted at 2026-09-29 10:25:44 on dist-test-slave-2rbl"
server_key: "03933d98b873216682bda9c276eaaf79"
server_key_iv: "8a574b412f52e6d90ff1c28eeb197111"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:46.539149 1914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:46.556522 1914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:46.559741 1914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:46.559948 1914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:46.567212 1914 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:46533
I20260929 10:25:46.567219 1982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:46533 every 8 connection(s)
I20260929 10:25:46.568267 1914 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:25:46.572516 1983 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:25:46.574230 1983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Bootstrap starting.
I20260929 10:25:46.575379 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 1914
I20260929 10:25:46.575829 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:42401
--local_ip_for_outbound_sockets=127.21.33.129
--tserver_master_addrs=127.21.33.190:46533
--webserver_port=42847
--webserver_interface=127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:46.590598 1983 log.cc:824] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:46.597385 1983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:25:46.597795 1983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Bootstrap complete.
I20260929 10:25:46.601989 1983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:46.602411 1983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba62ee211a94281adbcf3ca3052979d, State: Initialized, Role: FOLLOWER
I20260929 10:25:46.602944 1983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:46.603094 1983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:25:46.603183 1983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:25:46.603299 1983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:25:46.605151 1983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:46.605506 1983 leader_election.cc:302] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ba62ee211a94281adbcf3ca3052979d; no voters:
I20260929 10:25:46.605818 1983 leader_election.cc:288] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:25:46.605950 1987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:25:46.606201 1987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [term 2 LEADER]: Becoming Leader. State: Replica: 7ba62ee211a94281adbcf3ca3052979d, State: Running, Role: LEADER
I20260929 10:25:46.606500 1987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } }
I20260929 10:25:46.607100 1983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:46.607456 1987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ba62ee211a94281adbcf3ca3052979d. Latest consensus state: current_term: 2 leader_uuid: "7ba62ee211a94281adbcf3ca3052979d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } } }
I20260929 10:25:46.607553 1987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:46.608155 1988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7ba62ee211a94281adbcf3ca3052979d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba62ee211a94281adbcf3ca3052979d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46533 } } }
I20260929 10:25:46.608243 1988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:46.608501 1991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:25:46.611740 1991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:25:46.612321 1983 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 10:25:46.612342 1991 catalog_manager.cc:1259] Loaded cluster ID: 48b1e06849e94f0fbb295541279dfa13
I20260929 10:25:46.612530 1991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:25:46.614833 1991 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:25:46.615458 1991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Loaded TSK: 0
I20260929 10:25:46.616638 1991 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:25:46.713677 1985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:46.713987 1985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:46.714082 1985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:46.718827 1985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:46.718906 1985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:46.718993 1985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:25:46.726351 1985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:42401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=42847
--tserver_master_addrs=127.21.33.190:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:46.727521 1985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:46.728757 1985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:46.738710 2021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:46.738729 2023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:46.739217 2020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:46.739320 1985 server_base.cc:1034] running on GCE node
I20260929 10:25:46.739804 1985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:46.740562 1985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:46.741740 1985 hybrid_clock.cc:648] HybridClock initialized: now 1790677546741702 us; error 47 us; skew 500 ppm
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:46.745019 1985 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:25:46.746042 1985 webserver.cc:460] Webserver started at http://127.21.33.129:42847/ using document root <none> and password file <none>
I20260929 10:25:46.746658 1985 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:46.746765 1985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:46.773501 1985 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:25:46.777057 2031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:46.779726 1985 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 10:25:46.779989 1985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "68cd43362cba488f830e9a1dd49a6d2f"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "e53bb627114675bf9db7fdfd8429ff32"
server_key_iv: "f93f55e41b7c49414b1496cf83b4c8d0"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:46.781057 1985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:46.825029 1985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:46.828555 1985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:46.828778 1985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:46.829577 1985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:46.830618 1985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:46.830689 1985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:46.830756 1985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:46.830782 1985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:46.841853 1985 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:42401
I20260929 10:25:46.843268 1985 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:25:46.845388 2144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:42401 every 8 connection(s)
I20260929 10:25:46.846086 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 1985
I20260929 10:25:46.846531 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:42637
--local_ip_for_outbound_sockets=127.21.33.130
--tserver_master_addrs=127.21.33.190:46533
--webserver_port=43299
--webserver_interface=127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:25:46 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:46.867133 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:46.848238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46533 (local address 127.21.33.129:47875)
0929 10:25:46.848817 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:46.848880 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:46.850016 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:46.852245 (+ 2229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:46.852254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:46.852596 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:46.853286 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:46.853305 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.855581 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.855585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:46.857000 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:46.857012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.858104 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.858109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:46.858299 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:46.858979 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:46.858999 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:46.860849 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:46.863456 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:46.863462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:46.863477 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:46.863824 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:46.864268 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:46.864271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:46.864273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:46.864447 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:46.865233 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:46.865240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:46.865535 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 10:25:46.865719 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 10:25:46.866051 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":148,"threads_started":1}
I20260929 10:25:46.868832 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:46.849484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:47875 (local address 127.21.33.190:46533)
0929 10:25:46.850926 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:46.850950 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:46.850999 (+ 49us) server_negotiation.cc:407] Connection header received
0929 10:25:46.851371 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:46.851383 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:46.851760 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:46.853526 (+ 1766us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:46.853888 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.855422 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.857169 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:46.857910 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:46.858705 (+ 795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:46.861084 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:46.861091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:46.861108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:46.861140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:46.863314 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:46.863981 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:46.863988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:46.863993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:46.864149 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:46.864570 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:46.864573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:46.864575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:46.865059 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:46.867706 (+ 2647us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:46.867836 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 10:25:46.868076 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1284,"thread_start_us":124,"threads_started":1}
I20260929 10:25:46.871531 2145 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46533
I20260929 10:25:46.871799 2145 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:46.872414 2145 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
W20260929 10:25:46.874774 2145 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46533 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:25:46.874890 2145 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:46.875058 2145 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:46.875905 2145 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:46.876065 2145 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
W20260929 10:25:46.876771 2145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:25:46.995714 2147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:46.996064 2147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:46.996148 2147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:47.001830 2147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:47.001922 2147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:47.002017 2147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:25:47.008517 2147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:42637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=43299
--tserver_master_addrs=127.21.33.190:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:47.009720 2147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:47.010802 2147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:47.020692 2155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:47.020692 2157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:47.020959 2147 server_base.cc:1034] running on GCE node
W20260929 10:25:47.021086 2154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:47.021580 2147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:47.022370 2147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:47.023578 2147 hybrid_clock.cc:648] HybridClock initialized: now 1790677547023549 us; error 45 us; skew 500 ppm
Sep 29 10:25:47 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:47.026791 2147 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:25:47.027812 2147 webserver.cc:460] Webserver started at http://127.21.33.130:43299/ using document root <none> and password file <none>
I20260929 10:25:47.028383 2147 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:47.028499 2147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:25:47 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:47.068356 2147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:25:47.072505 2165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:47.074123 2147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 10:25:47.074270 2147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "682ced8e05c94f5b845d44c54342f781"
format_stamp: "Formatted at 2026-09-29 10:25:45 on dist-test-slave-2rbl"
server_key: "b3ae4679c9854a3983592b9f5b631f8d"
server_key_iv: "ae10a5ff5db57d9feb290f47b8ab1944"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:47.074862 2147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:47.089366 2147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:47.093468 2147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:47.093674 2147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:47.094386 2147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:47.095417 2147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:47.095512 2147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:47.095580 2147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:47.095634 2147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:47.106395 2147 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:42637
I20260929 10:25:47.107630 2147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:25:47.117025 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 2147
I20260929 10:25:47.117535 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.33.148:36227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:46215
--local_ip_for_outbound_sockets=127.21.33.131
--tserver_master_addrs=127.21.33.190:46533
--webserver_port=35911
--webserver_interface=127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:47.117870 2278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:42637 every 8 connection(s)
Sep 29 10:25:47 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:47.125566 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:47.112355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:50133 (local address 127.21.33.190:46533)
0929 10:25:47.112469 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:47.112473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:47.112501 (+ 28us) server_negotiation.cc:407] Connection header received
0929 10:25:47.112986 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:47.112990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:47.113045 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:47.113129 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:47.114214 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.114755 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.115862 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.116098 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.116207 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:47.121005 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:47.121011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:47.121015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:47.121044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:47.122912 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:47.123574 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:47.123579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:47.123580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:47.123672 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:47.124113 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:47.124117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:47.124119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:47.124355 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:47.124442 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:47.125184 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 10:25:47.125390 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:25:47.127348 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:47.111354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46533 (local address 127.21.33.130:50133)
0929 10:25:47.111820 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:47.111872 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:47.112781 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:47.113330 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:47.113337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:47.113603 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:47.114070 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:47.114083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.114882 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.114892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:47.115738 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:47.115747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.116241 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.116245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:47.116408 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:47.119023 (+ 2615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:47.119047 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:47.120794 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:47.123061 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:47.123068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:47.123088 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:47.123449 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:47.123792 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:47.123795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:47.123797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:47.124000 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:47.124532 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:47.124539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:47.124874 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 10:25:47.125822 (+ 948us) client_negotiation.cc:240] Negotiation successful
0929 10:25:47.126178 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260929 10:25:47.128854 2279 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46533
I20260929 10:25:47.129123 2279 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:47.129765 2279 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
W20260929 10:25:47.131444 2279 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46533 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:25:47.131578 2279 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:47.131763 2279 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:47.132730 2279 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:47.132908 2279 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
W20260929 10:25:47.133752 2279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:25:47.253027 2283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:47.253331 2283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:47.253400 2283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:47.257819 2283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:47.257902 2283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:47.257984 2283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:25:47.264309 2283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.33.148:36227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:46215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=35911
--tserver_master_addrs=127.21.33.190:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:47.265552 2283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:47.266603 2283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:47.277518 2288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:47.278911 2289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:47.279388 2283 server_base.cc:1034] running on GCE node
W20260929 10:25:47.279503 2291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:47.279898 2283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:47.280661 2283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:47.281827 2283 hybrid_clock.cc:648] HybridClock initialized: now 1790677547281798 us; error 46 us; skew 500 ppm
Sep 29 10:25:47 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:47.284538 2283 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:25:47.285594 2283 webserver.cc:460] Webserver started at http://127.21.33.131:35911/ using document root <none> and password file <none>
I20260929 10:25:47.286280 2283 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:47.286414 2283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 10:25:47 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:47.306667 2283 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 10:25:47.314047 2299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:47.315881 2283 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260929 10:25:47.316025 2283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "1c40180790274e8aa6e88fd2ebe7286f"
format_stamp: "Formatted at 2026-09-29 10:25:46 on dist-test-slave-2rbl"
server_key: "7c0d5ff1790e24c7bc0fa949c8fd03b5"
server_key_iv: "b371c3f7f2c5ea4d1c444cda773fc64b"
server_key_version: "kuduclusterkey@0"
I20260929 10:25:47.316816 2283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:47.345343 2283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:47.348913 2283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:47.349126 2283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:47.349901 2283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:47.351060 2283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:47.351128 2283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:47.351240 2283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:47.351282 2283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:47.362694 2283 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:46215
I20260929 10:25:47.362767 2412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:46215 every 8 connection(s)
I20260929 10:25:47.364156 2283 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:25:47.369117 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 2283
Sep 29 10:25:47 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:47.386828 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:47.369791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.21.33.190:46533)
0929 10:25:47.370373 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:47.370382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:47.370666 (+ 284us) server_negotiation.cc:407] Connection header received
0929 10:25:47.370860 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:47.370863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:47.370913 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:47.370976 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:47.371990 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.372846 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.374127 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.374360 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.374469 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:47.379194 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:47.379201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:47.379207 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:47.379258 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:47.381829 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:47.383645 (+ 1816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:47.383649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:47.383651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:47.383743 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:47.385137 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:47.385140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:47.385142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:47.385470 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:47.385674 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:47.386400 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 10:25:47.386629 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":507,"thread_start_us":92,"threads_started":1}
I20260929 10:25:47.389128 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:47.367848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46533 (local address 127.21.33.131:44779)
0929 10:25:47.368409 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:47.368471 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:47.369694 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:47.370673 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:47.370683 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:47.371050 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:47.371755 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:47.371773 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.372987 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.372991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:47.374440 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:47.374452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.375229 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.375233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:47.375442 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:47.376068 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:47.376089 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:47.377803 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:47.381969 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:47.381975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:47.381993 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:47.382350 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:47.382855 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:47.382859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:47.382861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:47.383072 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:47.384266 (+ 1194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:47.384273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:47.384646 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 10:25:47.387620 (+ 2974us) client_negotiation.cc:240] Negotiation successful
0929 10:25:47.388010 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":151,"threads_started":1}
I20260929 10:25:47.387543 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:47.367326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:44779 (local address 127.21.33.190:46533)
0929 10:25:47.367487 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:47.367491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:47.368593 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 10:25:47.370193 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:47.370197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:47.370251 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:47.370342 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:47.371906 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.372846 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.374598 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:47.374952 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:47.375034 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:47.377992 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:47.377997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:47.378000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:47.378026 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:47.379871 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:47.382475 (+ 2604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:47.382479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:47.382480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:47.382681 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:47.383866 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:47.383869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:47.383870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:47.384105 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:47.384194 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:47.387245 (+ 3051us) server_negotiation.cc:299] Negotiation successful
0929 10:25:47.387383 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":64}
I20260929 10:25:47.391369 2413 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46533
I20260929 10:25:47.391638 2413 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:47.392196 2413 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
W20260929 10:25:47.393772 2413 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46533 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 10:25:47.393896 2413 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:47.394065 2413 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:47.394941 2413 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:47.395102 2413 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
W20260929 10:25:47.395809 2413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 10:25:47 dist-test-slave-2rbl krb5kdc[951](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:47.877005 2145 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:47.877332 2145 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:47.878437 1947 ts_manager.cc:194] Registered new tserver with Master: 68cd43362cba488f830e9a1dd49a6d2f (127.21.33.129:42401)
I20260929 10:25:47.880139 1947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:47875
I20260929 10:25:48.133999 2279 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:48.134302 2279 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:48.135196 1947 ts_manager.cc:194] Registered new tserver with Master: 682ced8e05c94f5b845d44c54342f781 (127.21.33.130:42637)
I20260929 10:25:48.135963 1947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:50133
Sep 29 10:25:48 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
Sep 29 10:25:48 dist-test-slave-2rbl krb5kdc[951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677547, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for HTTP/127.21.33.148@KRBTEST.COM
I20260929 10:25:48.396061 2413 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:48.396420 2413 heartbeater.cc:511] Master 127.21.33.190:46533 requested a full tablet report, sending...
I20260929 10:25:48.397477 1947 ts_manager.cc:194] Registered new tserver with Master: 1c40180790274e8aa6e88fd2ebe7286f (127.21.33.131:46215)
I20260929 10:25:48.398310 1947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:44779
I20260929 10:25:48.402549 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:25:48.417603 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.405956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.21.33.190:46533)
0929 10:25:48.406867 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:48.406872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:48.406888 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:25:48.406936 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:48.406939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:48.406992 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:48.408194 (+ 1202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:48.408309 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.409305 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.410688 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.411063 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.411621 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.413025 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:48.413029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:48.413033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:48.413066 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:48.415076 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:48.415637 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:48.415654 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:48.415656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:48.415745 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:48.416073 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:48.416076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:48.416078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:48.416309 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:48.417162 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:48.417270 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 10:25:48.417411 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":115,"threads_started":1}
I20260929 10:25:48.422276 1947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:25:48.427264 1947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:25:48.493567 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.466968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:46215 (local address 127.0.0.1:44710)
0929 10:25:48.477668 (+ 10700us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:48.477680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:48.477750 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:48.478794 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:48.478800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:48.478825 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:48.479001 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.479010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.489989 (+ 10979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.489993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.491488 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.491501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.492954 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.492958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.493084 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.493124 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 10:25:48.493260 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 10:25:48.493397 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10649,"thread_start_us":81,"threads_started":1}
I20260929 10:25:48.496924 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.467225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.21.33.131:46215)
0929 10:25:48.477997 (+ 10772us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:48.478005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:48.478032 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:25:48.478116 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:48.478123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:48.478378 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:48.478541 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:48.479162 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.481505 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.491680 (+ 10175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.492634 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.492741 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.492893 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:48.496652 (+ 3759us) server_negotiation.cc:299] Negotiation successful
0929 10:25:48.496777 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10650,"thread_start_us":109,"threads_started":1}
I20260929 10:25:48.498560 2444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.466328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:42637 (local address 127.0.0.1:51830)
0929 10:25:48.477375 (+ 11047us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:48.477391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:48.477490 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:48.482320 (+ 4830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:48.482323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:48.482337 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:48.482412 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.482417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.494217 (+ 11800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.494221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.495386 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.495396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.498185 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.498189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.498281 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.498307 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:25:48.498354 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 10:25:48.498420 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10967,"thread_start_us":92,"threads_started":1}
I20260929 10:25:48.499491 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.466498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.21.33.130:42637)
0929 10:25:48.480589 (+ 14091us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:48.480596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:48.481623 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 10:25:48.481707 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:48.481713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:48.481911 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:48.484020 (+ 2109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:48.484159 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.485295 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.495644 (+ 10349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.496488 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.499046 (+ 2558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.499212 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:48.499262 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 10:25:48.499359 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13984,"thread_start_us":137,"threads_started":1}
I20260929 10:25:48.500701 2347 tablet_service.cc:1467] Processing CreateTablet for tablet 65b86105e6a943b7b59f6eb50669b02c (DEFAULT_TABLE table=test-table [id=287a7dc228ed4d4ab491b46c3a6134ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:48.503647 2207 tablet_service.cc:1467] Processing CreateTablet for tablet 65b86105e6a943b7b59f6eb50669b02c (DEFAULT_TABLE table=test-table [id=287a7dc228ed4d4ab491b46c3a6134ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:48.504848 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b86105e6a943b7b59f6eb50669b02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:48.506863 2347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b86105e6a943b7b59f6eb50669b02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:48.517294 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.465178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42401 (local address 127.0.0.1:43848)
0929 10:25:48.471595 (+ 6417us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:48.471671 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:48.471810 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:48.482071 (+ 10261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:48.482074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:48.482093 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:48.482203 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.482210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.495601 (+ 13391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.495605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.496775 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.496787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.516741 (+ 19954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.516746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.516878 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.516913 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 10:25:48.516983 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 10:25:48.517079 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6216,"thread_start_us":113,"threads_started":1}
I20260929 10:25:48.521900 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.465345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.21.33.129:42401)
0929 10:25:48.479976 (+ 14631us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:48.479983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:48.480008 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:25:48.480085 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:48.480091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:48.480310 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:48.482560 (+ 2250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:48.482712 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.483854 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.496953 (+ 13099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.497912 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.521371 (+ 23459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.521560 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:48.521632 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 10:25:48.521747 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14514,"thread_start_us":116,"threads_started":1}
I20260929 10:25:48.526584 2079 tablet_service.cc:1467] Processing CreateTablet for tablet 65b86105e6a943b7b59f6eb50669b02c (DEFAULT_TABLE table=test-table [id=287a7dc228ed4d4ab491b46c3a6134ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:48.527776 2079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b86105e6a943b7b59f6eb50669b02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:48.529561 2448 tablet_bootstrap.cc:492] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Bootstrap starting.
I20260929 10:25:48.531960 2448 tablet_bootstrap.cc:654] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:48.532876 2448 log.cc:824] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:48.535460 2448 tablet_bootstrap.cc:492] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: No bootstrap required, opened a new log
I20260929 10:25:48.535666 2448 ts_tablet_manager.cc:1397] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:25:48.539489 2448 raft_consensus.cc:348] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.539741 2448 raft_consensus.cc:374] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:48.539784 2448 raft_consensus.cc:729] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682ced8e05c94f5b845d44c54342f781, State: Initialized, Role: FOLLOWER
I20260929 10:25:48.540503 2448 consensus_queue.cc:260] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.540649 2451 tablet_bootstrap.cc:492] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Bootstrap starting.
I20260929 10:25:48.541651 2448 ts_tablet_manager.cc:1428] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:25:48.542084 2279 heartbeater.cc:503] Master 127.21.33.190:46533 was elected leader, sending a full tablet report...
I20260929 10:25:48.542717 2450 tablet_bootstrap.cc:492] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f: Bootstrap starting.
I20260929 10:25:48.546482 2450 tablet_bootstrap.cc:654] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:48.547380 2451 tablet_bootstrap.cc:654] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:48.548306 2451 log.cc:824] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:48.549602 2450 log.cc:824] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:48.551561 2451 tablet_bootstrap.cc:492] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: No bootstrap required, opened a new log
I20260929 10:25:48.551841 2451 ts_tablet_manager.cc:1397] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 10:25:48.553406 2450 tablet_bootstrap.cc:492] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f: No bootstrap required, opened a new log
I20260929 10:25:48.553629 2450 ts_tablet_manager.cc:1397] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260929 10:25:48.555704 2451 raft_consensus.cc:348] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.555991 2451 raft_consensus.cc:374] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:48.556052 2451 raft_consensus.cc:729] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c40180790274e8aa6e88fd2ebe7286f, State: Initialized, Role: FOLLOWER
I20260929 10:25:48.556598 2451 consensus_queue.cc:260] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.557761 2451 ts_tablet_manager.cc:1428] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:25:48.557890 2450 raft_consensus.cc:348] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.558202 2413 heartbeater.cc:503] Master 127.21.33.190:46533 was elected leader, sending a full tablet report...
I20260929 10:25:48.558246 2450 raft_consensus.cc:374] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:48.558312 2450 raft_consensus.cc:729] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cd43362cba488f830e9a1dd49a6d2f, State: Initialized, Role: FOLLOWER
I20260929 10:25:48.558840 2450 consensus_queue.cc:260] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.559942 2450 ts_tablet_manager.cc:1428] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260929 10:25:48.560280 2145 heartbeater.cc:503] Master 127.21.33.190:46533 was elected leader, sending a full tablet report...
W20260929 10:25:48.602151 2146 tablet.cc:2369] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:25:48.616549 2414 tablet.cc:2369] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:25:48.619958 2280 tablet.cc:2369] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:25:48.673362 2456 raft_consensus.cc:482] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:25:48.673660 2456 raft_consensus.cc:504] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.675339 2456 leader_election.cc:288] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 682ced8e05c94f5b845d44c54342f781 (127.21.33.130:42637), 1c40180790274e8aa6e88fd2ebe7286f (127.21.33.131:46215)
I20260929 10:25:48.679641 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.675741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:53961 (local address 127.21.33.130:42637)
0929 10:25:48.676134 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:48.676139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:48.676196 (+ 57us) server_negotiation.cc:407] Connection header received
0929 10:25:48.676253 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:48.676257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:48.676305 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:48.676385 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:48.676697 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.677530 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.678645 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.679037 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.679204 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.679257 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:48.679442 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 10:25:48.679512 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260929 10:25:48.679665 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.675577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:42637 (local address 127.21.33.129:53961)
0929 10:25:48.675957 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:48.675971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:48.676114 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:48.676395 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:48.676398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:48.676414 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:48.676575 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.676583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.677660 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.677664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.678503 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.678512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.679204 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.679208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.679308 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.679334 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:25:48.679425 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:25:48.679495 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":12,"thread_start_us":120,"threads_started":1}
I20260929 10:25:48.680322 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b86105e6a943b7b59f6eb50669b02c" candidate_uuid: "68cd43362cba488f830e9a1dd49a6d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682ced8e05c94f5b845d44c54342f781" is_pre_election: true
I20260929 10:25:48.680626 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.675593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:46215 (local address 127.21.33.129:43645)
0929 10:25:48.676228 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:48.676239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:48.676346 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:48.676722 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:48.676724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:48.676737 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:48.676821 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.676825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.677692 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.677696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.678990 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:48.679001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.680125 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.680128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:48.680283 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.680305 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:25:48.680383 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:25:48.680485 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":103,"threads_started":1}
I20260929 10:25:48.680626 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.675968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:43645 (local address 127.21.33.131:46215)
0929 10:25:48.676085 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:48.676090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:48.676313 (+ 223us) server_negotiation.cc:407] Connection header received
0929 10:25:48.676467 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:48.676474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:48.676518 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:48.676568 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:48.676927 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.677564 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.679140 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.679739 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.679837 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.679896 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:48.680417 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 10:25:48.680501 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:25:48.680665 2224 raft_consensus.cc:2454] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cd43362cba488f830e9a1dd49a6d2f in term 0.
I20260929 10:25:48.681192 2032 leader_election.cc:302] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682ced8e05c94f5b845d44c54342f781, 68cd43362cba488f830e9a1dd49a6d2f; no voters:
I20260929 10:25:48.681437 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b86105e6a943b7b59f6eb50669b02c" candidate_uuid: "68cd43362cba488f830e9a1dd49a6d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c40180790274e8aa6e88fd2ebe7286f" is_pre_election: true
I20260929 10:25:48.681504 2456 raft_consensus.cc:2781] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:25:48.681603 2456 raft_consensus.cc:482] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:25:48.681655 2456 raft_consensus.cc:3037] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:48.681736 2349 raft_consensus.cc:2454] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cd43362cba488f830e9a1dd49a6d2f in term 0.
I20260929 10:25:48.683000 2456 raft_consensus.cc:504] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.683398 2456 leader_election.cc:288] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [CANDIDATE]: Term 1 election: Requested vote from peers 682ced8e05c94f5b845d44c54342f781 (127.21.33.130:42637), 1c40180790274e8aa6e88fd2ebe7286f (127.21.33.131:46215)
I20260929 10:25:48.683777 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b86105e6a943b7b59f6eb50669b02c" candidate_uuid: "68cd43362cba488f830e9a1dd49a6d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682ced8e05c94f5b845d44c54342f781"
I20260929 10:25:48.683907 2224 raft_consensus.cc:3037] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:48.685616 2349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b86105e6a943b7b59f6eb50669b02c" candidate_uuid: "68cd43362cba488f830e9a1dd49a6d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c40180790274e8aa6e88fd2ebe7286f"
I20260929 10:25:48.685755 2349 raft_consensus.cc:3037] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:48.685873 2224 raft_consensus.cc:2454] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cd43362cba488f830e9a1dd49a6d2f in term 1.
I20260929 10:25:48.686158 2032 leader_election.cc:302] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682ced8e05c94f5b845d44c54342f781, 68cd43362cba488f830e9a1dd49a6d2f; no voters:
I20260929 10:25:48.686352 2456 raft_consensus.cc:2781] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:48.686604 2456 raft_consensus.cc:686] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [term 1 LEADER]: Becoming Leader. State: Replica: 68cd43362cba488f830e9a1dd49a6d2f, State: Running, Role: LEADER
I20260929 10:25:48.686965 2456 consensus_queue.cc:237] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } }
I20260929 10:25:48.687148 2349 raft_consensus.cc:2454] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cd43362cba488f830e9a1dd49a6d2f in term 1.
I20260929 10:25:48.690511 1946 catalog_manager.cc:5495] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 68cd43362cba488f830e9a1dd49a6d2f (127.21.33.129). New cstate: current_term: 1 leader_uuid: "68cd43362cba488f830e9a1dd49a6d2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cd43362cba488f830e9a1dd49a6d2f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 42401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 } health_report { overall_health: UNKNOWN } } }
I20260929 10:25:48.713903 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.709579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.21.33.129:42401)
0929 10:25:48.709725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:48.709729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:48.709745 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:25:48.709865 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:48.709867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:48.709931 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:48.710021 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:25:48.710345 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.710858 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.712264 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:48.712476 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:48.712561 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:48.712915 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:25:48.713434 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:25:48.713530 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:48.713653 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 10:25:48.713706 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:25:48.719785 2349 raft_consensus.cc:1261] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 1 FOLLOWER]: Refusing update from remote peer 68cd43362cba488f830e9a1dd49a6d2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:48.719873 2224 raft_consensus.cc:1261] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 1 FOLLOWER]: Refusing update from remote peer 68cd43362cba488f830e9a1dd49a6d2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:48.720556 2456 consensus_queue.cc:1035] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "682ced8e05c94f5b845d44c54342f781" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:48.720782 2466 consensus_queue.cc:1035] T 65b86105e6a943b7b59f6eb50669b02c P 68cd43362cba488f830e9a1dd49a6d2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c40180790274e8aa6e88fd2ebe7286f" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 46215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:48.728993 2469 mvcc.cc:205] Tried to move back new op lower bound from 7334615239549026304 to 7334615239424851968. Current Snapshot: MvccSnapshot[applied={T|T < 7334615239549026304}]
I20260929 10:25:48.730585 2471 mvcc.cc:205] Tried to move back new op lower bound from 7334615239549026304 to 7334615239424851968. Current Snapshot: MvccSnapshot[applied={T|T < 7334615239549026304}]
I20260929 10:25:48.734225 2470 mvcc.cc:205] Tried to move back new op lower bound from 7334615239549026304 to 7334615239424851968. Current Snapshot: MvccSnapshot[applied={T|T < 7334615239549026304}]
I20260929 10:25:48.740785 1946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:48.740957 1946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:48.743705 1946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ba62ee211a94281adbcf3ca3052979d: Sending DeleteTablet for 3 replicas of tablet 65b86105e6a943b7b59f6eb50669b02c
I20260929 10:25:48.744580 2347 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b86105e6a943b7b59f6eb50669b02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:48 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:44710
I20260929 10:25:48.744801 2207 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b86105e6a943b7b59f6eb50669b02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:48 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:51830
I20260929 10:25:48.745328 2480 tablet_replica.cc:330] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: stopping tablet replica
I20260929 10:25:48.745792 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 1985
I20260929 10:25:48.745923 2079 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b86105e6a943b7b59f6eb50669b02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:48 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:43848
I20260929 10:25:48.746857 2479 tablet_replica.cc:330] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: stopping tablet replica
I20260929 10:25:48.747215 2479 raft_consensus.cc:2229] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:48.747540 2479 raft_consensus.cc:2258] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:48.748692 2480 raft_consensus.cc:2229] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:48.749084 2480 raft_consensus.cc:2258] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:48.749823 2479 ts_tablet_manager.cc:1905] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:48.750712 2480 ts_tablet_manager.cc:1905] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:48.752712 2479 ts_tablet_manager.cc:1918] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:48.752813 2479 log.cc:1196] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/65b86105e6a943b7b59f6eb50669b02c
I20260929 10:25:48.753157 2479 ts_tablet_manager.cc:1939] T 65b86105e6a943b7b59f6eb50669b02c P 1c40180790274e8aa6e88fd2ebe7286f: Deleting consensus metadata
W20260929 10:25:48.753455 1934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:42401 (error 108)
I20260929 10:25:48.754107 2480 ts_tablet_manager.cc:1918] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:48.754273 2480 log.cc:1196] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/65b86105e6a943b7b59f6eb50669b02c
I20260929 10:25:48.754477 1933 catalog_manager.cc:4841] TS 1c40180790274e8aa6e88fd2ebe7286f (127.21.33.131:46215): tablet 65b86105e6a943b7b59f6eb50669b02c (table test-table [id=287a7dc228ed4d4ab491b46c3a6134ee]) successfully deleted
I20260929 10:25:48.754719 2480 ts_tablet_manager.cc:1939] T 65b86105e6a943b7b59f6eb50669b02c P 682ced8e05c94f5b845d44c54342f781: Deleting consensus metadata
I20260929 10:25:48.754917 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.754602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:42401 (local address 127.0.0.1:43868)
0929 10:25:48.754774 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:48.754838 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 10:25:48.755095 1934 catalog_manager.cc:4584] TS 68cd43362cba488f830e9a1dd49a6d2f (127.21.33.129:42401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65b86105e6a943b7b59f6eb50669b02c: Network error: Client connection negotiation failed: client connection to 127.21.33.129:42401: connect: Connection refused (error 111)
I20260929 10:25:48.755396 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 2147
I20260929 10:25:48.761071 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 2283
I20260929 10:25:48.762688 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:48.761305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:42637 (local address 127.0.0.1:51842)
0929 10:25:48.762487 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:48.762572 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.130:42637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1075}
I20260929 10:25:48.766939 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 1914
I20260929 10:25:48.775413 21638 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 10:25:49.110489 21638 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 10:25:49.110596 21638 mini_ranger.cc:66] Stopping Ranger...
I20260929 10:25:49.455659 21638 mini_ranger.cc:68] Stopped Ranger
2026-09-29 10:25:49.455 UTC [981] LOG: received smart shutdown request
2026-09-29 10:25:49.459 UTC [981] LOG: background worker "logical replication launcher" (PID 990) exited with exit code 1
2026-09-29 10:25:49.459 UTC [985] LOG: shutting down
2026-09-29 10:25:49.637 UTC [981] LOG: database system is shut down
2026-09-29T10:25:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48621 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:25:49 dist-test-slave-2rbl krb5kdc[2490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:25:49 dist-test-slave-2rbl krb5kdc[2490](info): set up 2 sockets
Sep 29 10:25:49 dist-test-slave-2rbl krb5kdc[2490](info): commencing operation
krb5kdc: starting...
W20260929 10:25:50.797894 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:25:50 dist-test-slave-2rbl krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:25:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:25:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:25:50.920568 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:34769
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:34769,127.21.33.189:37705,127.21.33.188:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:51.029719 2503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:51.029989 2503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:51.030066 2503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:51.034168 2503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:51.034250 2503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:51.034283 2503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:51.034338 2503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:51.034374 2503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:51.039670 2503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:34769,127.21.33.189:37705,127.21.33.188:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:34769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:51.040694 2503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:51.041738 2503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:51.050382 2511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:51.050381 2508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:51.050612 2503 server_base.cc:1034] running on GCE node
W20260929 10:25:51.050387 2509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:51.051271 2503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:51.052320 2503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:51.053511 2503 hybrid_clock.cc:648] HybridClock initialized: now 1790677551053476 us; error 45 us; skew 500 ppm
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:51.056454 2503 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:25:51.057389 2503 webserver.cc:460] Webserver started at http://127.21.33.190:42359/ using document root <none> and password file <none>
I20260929 10:25:51.057921 2503 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:51.057986 2503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:51.058154 2503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:51.060184 2503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "8968cc1e0ecc47f697dd833587033cd4"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "cfcc9013a44d551c43b48ee99e04d671"
server_key_iv: "96885385a2612f6f7e93224ab64edfad"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.060729 2503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "8968cc1e0ecc47f697dd833587033cd4"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "cfcc9013a44d551c43b48ee99e04d671"
server_key_iv: "96885385a2612f6f7e93224ab64edfad"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.064675 2503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 10:25:51.067586 2518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:51.068989 2503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:25:51.069103 2503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "8968cc1e0ecc47f697dd833587033cd4"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "cfcc9013a44d551c43b48ee99e04d671"
server_key_iv: "96885385a2612f6f7e93224ab64edfad"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.069275 2503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:51.082104 2503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:51.085018 2503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:51.085170 2503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:51.092080 2503 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:34769
I20260929 10:25:51.092098 2570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:34769 every 8 connection(s)
I20260929 10:25:51.093021 2503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:25:51.095769 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:51.096844 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 2503
I20260929 10:25:51.096989 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:25:51.097328 21638 external_mini_cluster.cc:1428] Setting key e5e6ba398e677f36699ea4c3b42efc5b
I20260929 10:25:51.100136 2571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.33.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.189@KRBTEST.COM" created.
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:51.117924 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.102888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.21.33.190:34769)
0929 10:25:51.103496 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.103509 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.103555 (+ 46us) server_negotiation.cc:407] Connection header received
0929 10:25:51.104581 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.104587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.104831 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.105052 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.105862 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.107355 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.108951 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.109613 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.109742 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.112443 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.112453 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.112471 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.112503 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.114704 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.115198 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.115206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.115210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.115366 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.115688 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.115692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.115694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.116251 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.116400 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.116815 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.117103 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":484,"spinlock_wait_cycles":3840,"thread_start_us":186,"threads_started":1}
I20260929 10:25:51.117924 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.102888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34769 (local address 127.0.0.1:54376)
0929 10:25:51.103371 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.103423 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.104255 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.105055 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.105059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.105452 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.105706 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.105719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.107506 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.107509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.108796 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.108808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.109784 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.109791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.109964 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.110582 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.110608 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.112290 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.114833 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.114841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.114849 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.115086 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.115479 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.115481 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.115482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.115592 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.116396 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.116402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.116543 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.116701 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.116992 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":136,"threads_started":1}
I20260929 10:25:51.120442 2571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.121317 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.121008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43140)
0929 10:25:51.121126 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.121250 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 10:25:51.121500 2519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
I20260929 10:25:51.122294 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.122103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43150)
0929 10:25:51.122199 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.122241 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 10:25:51.122469 2571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:37705: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab.
Entry for principal kudu/127.21.33.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab.
I20260929 10:25:51.128546 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:37705
--webserver_interface=127.21.33.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab
--principal=kudu/127.21.33.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:34769,127.21.33.189:37705,127.21.33.188:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:51.171761 2571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } attempt: 1
I20260929 10:25:51.172801 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.172378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43164)
0929 10:25:51.172588 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.172682 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 10:25:51.173834 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.173584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43180)
0929 10:25:51.173700 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.173756 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 10:25:51.174064 2571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:37705: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
W20260929 10:25:51.242035 2578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260929 10:25:51.242255 2571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } attempt: 2
W20260929 10:25:51.242383 2578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:51.242441 2578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:25:51.243162 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.242817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43186)
0929 10:25:51.243002 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.243072 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 10:25:51.243983 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.243761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43188)
0929 10:25:51.243866 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.243915 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 10:25:51.244141 2571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:37705: Network error: Client connection negotiation failed: client connection to 127.21.33.189:37705: connect: Connection refused (error 111)
W20260929 10:25:51.247313 2578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:51.247372 2578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:51.247392 2578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:51.247411 2578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:51.247453 2578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:51.252923 2578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:34769,127.21.33.189:37705,127.21.33.188:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab
--principal=kudu/127.21.33.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:37705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.33.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:51.254087 2578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:51.255148 2578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:51.264015 2584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:51.264017 2583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:51.264015 2586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:51.264453 2578 server_base.cc:1034] running on GCE node
I20260929 10:25:51.265020 2578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:51.266243 2578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:51.267431 2578 hybrid_clock.cc:648] HybridClock initialized: now 1790677551267417 us; error 35 us; skew 500 ppm
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:51.270210 2578 init.cc:377] Logged in from keytab as kudu/127.21.33.189@KRBTEST.COM (short username kudu)
I20260929 10:25:51.271050 2578 webserver.cc:460] Webserver started at http://127.21.33.189:43521/ using document root <none> and password file <none>
I20260929 10:25:51.271605 2578 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:51.271689 2578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:51.271924 2578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:51.274075 2578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data/instance:
uuid: "6827007bbe0446849dd7257a2dfc3cf8"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "0e75b92719b1fa7838e898f31d5c83d8"
server_key_iv: "8e5f651cf65010c7de6cc38ddd71beed"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.274632 2578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/wal/instance:
uuid: "6827007bbe0446849dd7257a2dfc3cf8"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "0e75b92719b1fa7838e898f31d5c83d8"
server_key_iv: "8e5f651cf65010c7de6cc38ddd71beed"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.278266 2578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:25:51.280583 2593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:51.281807 2578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:25:51.281935 2578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/wal
uuid: "6827007bbe0446849dd7257a2dfc3cf8"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "0e75b92719b1fa7838e898f31d5c83d8"
server_key_iv: "8e5f651cf65010c7de6cc38ddd71beed"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.282069 2578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:51.314023 2578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:51.316972 2578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:51.317190 2578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:51.324363 2578 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.189:37705
I20260929 10:25:51.324388 2645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.189:37705 every 8 connection(s)
I20260929 10:25:51.325388 2578 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
I20260929 10:25:51.328049 2646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:51.331709 2646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.335311 2571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } attempt: 3
I20260929 10:25:51.335475 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 2578
I20260929 10:25:51.335615 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-1/wal/instance
I20260929 10:25:51.335969 21638 external_mini_cluster.cc:1428] Setting key 245f930d339bd05212c2b2d93776a9f2
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.188@KRBTEST.COM; defaulting to no policy
I20260929 10:25:51.347400 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.335752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43190)
0929 10:25:51.335905 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.335922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.336010 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.336749 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.336752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.336945 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.337073 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.337080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.338726 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.338731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.339241 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.339248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.340109 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.340115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.340269 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.340805 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.340825 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.342467 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.345339 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.345346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.345350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.345648 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.346052 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.346055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.346057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.346154 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.346860 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.346866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.346992 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.347094 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.347228 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"spinlock_wait_cycles":7424}
I20260929 10:25:51.347995 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.333648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.21.33.190:34769)
0929 10:25:51.333799 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.333803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.334659 (+ 856us) server_negotiation.cc:407] Connection header received
0929 10:25:51.336154 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.336157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.336202 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.336256 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.337667 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.338411 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.339690 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.339963 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.340087 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.343316 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.343320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.343323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.343350 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.344923 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.345546 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.345552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.345557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.345663 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.346081 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.346084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.346085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.346339 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.346466 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.347666 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.347846 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:25:51.348448 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.334012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34769 (local address 127.0.0.1:54382)
0929 10:25:51.334493 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.334539 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.335597 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.336566 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.336578 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.337088 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.337508 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.337526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.338545 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.338551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.339563 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.339573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.340124 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.340128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.340296 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.341050 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.341076 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.343174 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.345061 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.345072 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.345083 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.345434 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.345786 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.345791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.345796 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.345970 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.346458 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.346464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.346743 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.346914 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.347273 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.188@KRBTEST.COM" created.
I20260929 10:25:51.348465 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.335821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.21.33.189:37705)
0929 10:25:51.336198 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.336207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.336239 (+ 32us) server_negotiation.cc:407] Connection header received
0929 10:25:51.336352 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.336365 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.336585 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.336769 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.337304 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.338615 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.339350 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.339966 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.340528 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.342633 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.342644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.342660 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.342692 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.345225 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.345785 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.345792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.345794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.345948 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.346249 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.346252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.346254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.346729 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.346870 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.347216 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.347409 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":2432,"thread_start_us":171,"threads_started":1}
I20260929 10:25:51.349947 2646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.350803 2571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.351518 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.351267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35254)
0929 10:25:51.351380 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.351436 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 10:25:51.352239 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.352008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35256)
0929 10:25:51.352137 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.352188 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 10:25:51.352406 2571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:44641: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:25:51.359243 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.350508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43206)
0929 10:25:51.350892 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.350919 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.351011 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.351296 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.351302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.351472 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.351590 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.351599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.352673 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.352676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.353262 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.353272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.353968 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.353972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.354061 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.354568 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.354587 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.355795 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.357816 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.357818 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.357820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.358023 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.358292 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.358294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.358295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.358369 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.358833 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.358835 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.358915 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.358996 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.359120 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1}
I20260929 10:25:51.359423 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.350560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.21.33.189:37705)
0929 10:25:51.350681 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.350686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.351094 (+ 408us) server_negotiation.cc:407] Connection header received
0929 10:25:51.351126 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.351130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.351184 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.351265 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.351736 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.352549 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.353392 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.353835 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.353968 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.355930 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.355934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.355945 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.355975 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.357708 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.358139 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.358142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.358143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.358215 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.358520 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.358524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.358525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.358736 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.358818 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.359051 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.359216 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":33}
I20260929 10:25:51.360146 2646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.360872 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.360651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35268)
0929 10:25:51.360743 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.360819 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 10:25:51.361570 2596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
I20260929 10:25:51.362576 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.362373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35284)
0929 10:25:51.362462 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.362511 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 10:25:51.362737 2646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:44641: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab.
Entry for principal kudu/127.21.33.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab.
I20260929 10:25:51.366639 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:44641
--webserver_interface=127.21.33.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab
--principal=kudu/127.21.33.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:41249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:34769,127.21.33.189:37705,127.21.33.188:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:51.383574 2571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } attempt: 1
I20260929 10:25:51.384469 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.384143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35292)
0929 10:25:51.384318 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.384382 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 10:25:51.385288 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.385009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35304)
0929 10:25:51.385115 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.385169 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 10:25:51.385422 2571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:44641: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
I20260929 10:25:51.412030 2646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } attempt: 1
I20260929 10:25:51.413015 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.412664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35308)
0929 10:25:51.412825 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.412898 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 10:25:51.413805 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.413582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35318)
0929 10:25:51.413694 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.413747 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 10:25:51.413956 2646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:44641: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
I20260929 10:25:51.460592 2571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } attempt: 2
I20260929 10:25:51.461519 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.461160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35330)
0929 10:25:51.461341 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.461401 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 10:25:51.462273 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.462076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35340)
0929 10:25:51.462176 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.462226 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 10:25:51.462396 2571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:44641: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
W20260929 10:25:51.478847 2653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:51.479141 2653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:51.479238 2653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:25:51.482144 2646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } attempt: 2
I20260929 10:25:51.482928 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.482646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35348)
0929 10:25:51.482785 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.482851 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 10:25:51.483448 2653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:51.483515 2653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:51.483563 2653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:51.483593 2653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:51.483628 2653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:51.483670 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.483432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35356)
0929 10:25:51.483536 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.483591 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 10:25:51.483811 2646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:44641: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
I20260929 10:25:51.489272 2653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:41249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:34769,127.21.33.189:37705,127.21.33.188:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab
--principal=kudu/127.21.33.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.33.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:51.490397 2653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:51.491460 2653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:51.499897 2658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:51.499894 2659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:51.500003 2653 server_base.cc:1034] running on GCE node
W20260929 10:25:51.499894 2661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:51.500749 2653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:51.501909 2653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:51.503106 2653 hybrid_clock.cc:648] HybridClock initialized: now 1790677551503072 us; error 45 us; skew 500 ppm
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:51.506023 2653 init.cc:377] Logged in from keytab as kudu/127.21.33.188@KRBTEST.COM (short username kudu)
I20260929 10:25:51.506880 2653 webserver.cc:460] Webserver started at http://127.21.33.188:33465/ using document root <none> and password file <none>
I20260929 10:25:51.507439 2653 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:51.507530 2653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:51.507753 2653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:51.509876 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data/instance:
uuid: "bca4f50f1d814974a64f388c1dd868a8"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "df29ff085b5297776eae1a66e99af1ab"
server_key_iv: "ba91cacc33a0a4b4c5d5972394cf0bbb"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.510429 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/wal/instance:
uuid: "bca4f50f1d814974a64f388c1dd868a8"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "df29ff085b5297776eae1a66e99af1ab"
server_key_iv: "ba91cacc33a0a4b4c5d5972394cf0bbb"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.514153 2653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:25:51.516569 2668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:51.517791 2653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:51.517925 2653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/wal
uuid: "bca4f50f1d814974a64f388c1dd868a8"
format_stamp: "Formatted at 2026-09-29 10:25:51 on dist-test-slave-2rbl"
server_key: "df29ff085b5297776eae1a66e99af1ab"
server_key_iv: "ba91cacc33a0a4b4c5d5972394cf0bbb"
server_key_version: "encryptionkey@0"
I20260929 10:25:51.518054 2653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:51.551971 2653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:51.555166 2653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:51.555373 2653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:51.561551 2571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } attempt: 3
I20260929 10:25:51.562410 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.562076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35366)
0929 10:25:51.562224 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.562298 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 10:25:51.562934 2653 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.188:44641
I20260929 10:25:51.563006 2720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.188:44641 every 8 connection(s)
I20260929 10:25:51.563982 2653 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
I20260929 10:25:51.566748 2722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:25:51.570528 2722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.573437 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 2653
I20260929 10:25:51.573560 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790677387835684-21638-0/minicluster-data/master-2/wal/instance
I20260929 10:25:51.573848 21638 external_mini_cluster.cc:1428] Setting key f503d5227178bd5d4484304cc3b0db81
I20260929 10:25:51.574702 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.562987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35378)
0929 10:25:51.563148 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.563163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.563257 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.565672 (+ 2415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.565674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.565863 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.565983 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.565991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.567407 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.567411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.567931 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.567938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.568638 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.568642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.568719 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.569226 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.569240 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.570399 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.572592 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.572595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.572597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.572797 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.573167 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.573172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.573174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.573312 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.574163 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.574168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.574269 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.574360 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.574509 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 10:25:51.575786 2646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } attempt: 3
I20260929 10:25:51.576134 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.563830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.21.33.188:44641)
0929 10:25:51.564223 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.564242 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.564287 (+ 45us) server_negotiation.cc:407] Connection header received
0929 10:25:51.564998 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.565016 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.565407 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.565745 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.566169 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.567286 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.568052 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.568520 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.568684 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.570595 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.570599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.570611 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.570634 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.572481 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.572908 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.572913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.572919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.573079 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.573506 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.573511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.573513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.574035 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.575022 (+ 987us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.575156 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.575364 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":145,"threads_started":1}
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:25:51.582473 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: Bootstrap starting.
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:25:51.585342 2571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:51.586347 2571 log.cc:824] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:51.587025 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.576200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35402)
0929 10:25:51.576376 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.576389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.576466 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.577926 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.577929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.578099 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.578219 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.578229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.579867 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.579873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.580588 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.580598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.581032 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.581036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.581130 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.581623 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.581637 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.582876 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.585140 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.585145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.585151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.585446 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.585791 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.585796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.585799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.585890 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.586461 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.586464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.586592 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.586679 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.586835 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 10:25:51.587664 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.574466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.21.33.188:44641)
0929 10:25:51.575095 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.575104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.575117 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:25:51.576198 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.576202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.576247 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.576322 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.577357 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.578119 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.579372 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.579686 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.579795 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.583178 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.583184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.583189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.583213 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.585094 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.585666 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.585671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.585726 (+ 55us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.585813 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.586113 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.586116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.586118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.586400 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.586661 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.587294 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.587460 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":565,"spinlock_wait_cycles":1920,"thread_start_us":97,"threads_started":1}
I20260929 10:25:51.587679 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.576632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.21.33.188:44641)
0929 10:25:51.577705 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.577713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.577727 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:25:51.577761 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.577764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.577798 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.578322 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.578417 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.579746 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.580707 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.580913 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.581022 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.583031 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.583036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.583039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.583061 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.585007 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.585589 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.585592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.585594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.585677 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.586004 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.586007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.586008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.586342 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.587231 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.587301 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.587510 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260929 10:25:51.588728 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.572169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.21.33.190:34769)
0929 10:25:51.572313 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.572317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.573086 (+ 769us) server_negotiation.cc:407] Connection header received
0929 10:25:51.573265 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.573269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.573314 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.573397 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.574094 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.574848 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.575768 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.576069 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.576178 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.578741 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.578745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.578748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.578780 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.585999 (+ 7219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.586626 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.586631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.586632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.586735 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.587033 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.587039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.587044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.587299 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.587393 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.588482 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.588597 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 10:25:51.588742 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.572149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:34769 (local address 127.0.0.1:54384)
0929 10:25:51.572927 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.572991 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.573137 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.573427 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.573433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.573810 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.573972 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.573980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.574974 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.574978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.575652 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.575661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.576287 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.576294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.576426 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.576992 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.577005 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.578603 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.586191 (+ 7588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.586203 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.586209 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.586489 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.586840 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.586842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.586844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.586924 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.588096 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.588101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.588243 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.588395 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.588584 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":128,"threads_started":1}
I20260929 10:25:51.589845 2722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.590308 2571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: No bootstrap required, opened a new log
I20260929 10:25:51.591034 2646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: Bootstrap starting.
I20260929 10:25:51.593652 2646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:51.594262 2571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.594480 2571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:51.594537 2571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8968cc1e0ecc47f697dd833587033cd4, State: Initialized, Role: FOLLOWER
I20260929 10:25:51.594563 2646 log.cc:824] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:51.595052 2571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:25:51.596099 2730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.596235 2730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:25:51.596490 2571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:51.597369 2646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: No bootstrap required, opened a new log
I20260929 10:25:51.600296 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.590336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:37705 (local address 127.0.0.1:43220)
0929 10:25:51.590461 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.590477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.590552 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.590897 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.590900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.591108 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.591219 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.591226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.592216 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.592220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.592969 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.592980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.593622 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.593626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.593729 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.594250 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.594264 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.595754 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.598340 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.598344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.598346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.598620 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.598961 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.598966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.598969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.599066 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.599569 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.599572 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.599665 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.599761 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.599891 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 10:25:51.601095 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.590585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.21.33.189:37705)
0929 10:25:51.590692 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.590696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.590709 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:25:51.590744 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.590747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.590786 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.590868 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.591342 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.592092 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.593103 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.593498 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.593605 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.595884 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.595887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.595890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.595913 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.598207 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.598756 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.598760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.598762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.598852 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.599176 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.599179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.599181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.599420 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.599527 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.600850 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.600962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:51.602304 2722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:25:51.602260 2503 master_runner.cc:186] Error getting master registration for 127.21.33.189:37705: OK, instance_id { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" instance_seqno: 1790677551322341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:25:51.606649 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.595507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.21.33.190:34769)
0929 10:25:51.595634 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.595639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.596387 (+ 748us) server_negotiation.cc:407] Connection header received
0929 10:25:51.596432 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.596434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.596495 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.596584 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.597369 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.598088 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.599245 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.599547 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.599637 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.602602 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.602608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.602612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.602640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.604430 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.604912 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.604918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.604920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.605001 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.605341 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.605345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.605346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.605556 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.605738 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.606384 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.606501 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:25:51.606684 2503 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:25:51.609949 2646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.610196 2646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:51.610260 2646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6827007bbe0446849dd7257a2dfc3cf8, State: Initialized, Role: FOLLOWER
W20260929 10:25:51.602278 2741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:25:51.610597 2741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:25:51.610960 2646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.612447 2646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 10:25:51 dist-test-slave-2rbl krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:25:51.615070 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.602835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:44641 (local address 127.0.0.1:35416)
0929 10:25:51.603357 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:51.603373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:51.603732 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:51.604067 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:51.604070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:51.604252 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:51.604341 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.604349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.605487 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.605490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.606371 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:51.606382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.607733 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.607736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:51.607820 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.608328 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:51.608344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:51.610093 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:51.612946 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.612949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.612951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.613233 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.613556 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:51.613559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:51.613562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:51.613647 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:51.614173 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:51.614177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:51.614261 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 10:25:51.614370 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:25:51.614541 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451}
I20260929 10:25:51.615657 2743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.615813 2743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:25:51.616317 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.602865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.21.33.188:44641)
0929 10:25:51.603019 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.603026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.603433 (+ 407us) server_negotiation.cc:407] Connection header received
0929 10:25:51.603851 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.603854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.603894 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.603958 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.604480 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.605350 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.606523 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.607504 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.607621 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.610429 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.610434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.610437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.610474 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.612824 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.613345 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.613349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.613351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.613453 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.613804 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.613808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.613810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.614037 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.614135 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.616032 (+ 1897us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.616161 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:25:51.618080 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.607782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.21.33.189:37705)
0929 10:25:51.607885 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.607889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.608093 (+ 204us) server_negotiation.cc:407] Connection header received
0929 10:25:51.608251 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.608254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.608297 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.608346 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.609224 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.609914 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.610950 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.611283 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.611358 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.614347 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.614352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.614358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.614390 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.616165 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.616641 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.616644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.616646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.616728 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.617071 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.617074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.617076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.617304 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.617455 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.617824 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.617933 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:25:51.619841 2722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8: Bootstrap starting.
I20260929 10:25:51.621335 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:25:51.623302 2722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:51.620253 2578 master_runner.cc:186] Error getting master registration for 127.21.33.188:44641: OK, instance_id { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" instance_seqno: 1790677551560470 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:25:51.624337 2578 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:25:51.624491 2722 log.cc:824] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8: Log is configured to *not* fsync() on all Append() calls
W20260929 10:25:51.626096 2754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:25:51.626169 2754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:25:51.627051 2722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8: No bootstrap required, opened a new log
I20260929 10:25:51.632211 2722 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.632562 2722 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:51.632663 2722 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca4f50f1d814974a64f388c1dd868a8, State: Initialized, Role: FOLLOWER
I20260929 10:25:51.633311 2722 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.634905 2722 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:51.641443 2764 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.641615 2764 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:25:51.642395 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.626585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.21.33.190:34769)
0929 10:25:51.626709 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.626713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.628270 (+ 1557us) server_negotiation.cc:407] Connection header received
0929 10:25:51.628462 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.628466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.628520 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.628576 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.630949 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.631769 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.635086 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.635435 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.635519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.638214 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.638218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.638222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.638254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.640139 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.640719 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.640723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.640725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.640813 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.641134 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.641138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.641140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.641430 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.641605 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.642090 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.642219 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:25:51.643746 2653 master_runner.cc:263] No leader master found from master bca4f50f1d814974a64f388c1dd868a8
I20260929 10:25:51.643808 2653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:25:51.649595 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.627000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.21.33.188:44641)
0929 10:25:51.627113 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.627117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.629753 (+ 2636us) server_negotiation.cc:407] Connection header received
0929 10:25:51.629919 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.629922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.629965 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.630017 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.633053 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.633860 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.636079 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.636420 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.636499 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.645150 (+ 8651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.645155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.645158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.645186 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.647267 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.647805 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.647809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.647812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.647900 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.648225 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.648229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.648230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.648495 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.648603 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.649302 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.649426 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 10:25:51.650398 2775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:25:51.650480 2775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:25:51.654127 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:51.627686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.21.33.189:37705)
0929 10:25:51.627788 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:51.627792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:51.644072 (+ 16280us) server_negotiation.cc:407] Connection header received
0929 10:25:51.645277 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:51.645280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:51.645328 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:51.645376 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:51.646143 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.646913 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.648010 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:51.648353 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:51.648422 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:51.650121 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:51.650125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:51.650127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:51.650158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:51.652331 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.652843 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.652846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.652848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.652931 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:51.653246 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:51.653249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:51.653251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:51.653453 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:51.653593 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:51.653841 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 10:25:51.653961 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:25:51.743106 2764 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:25:51.743340 2764 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.744547 2764 leader_election.cc:288] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8968cc1e0ecc47f697dd833587033cd4 (127.21.33.190:34769), 6827007bbe0446849dd7257a2dfc3cf8 (127.21.33.189:37705)
I20260929 10:25:51.745313 2620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca4f50f1d814974a64f388c1dd868a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6827007bbe0446849dd7257a2dfc3cf8" is_pre_election: true
I20260929 10:25:51.745666 2620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bca4f50f1d814974a64f388c1dd868a8 in term 0.
I20260929 10:25:51.746165 2669 leader_election.cc:302] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6827007bbe0446849dd7257a2dfc3cf8, bca4f50f1d814974a64f388c1dd868a8; no voters:
I20260929 10:25:51.746526 2764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:25:51.746621 2764 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:25:51.746680 2764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:51.748142 2764 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.749445 2544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca4f50f1d814974a64f388c1dd868a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8968cc1e0ecc47f697dd833587033cd4"
I20260929 10:25:51.749728 2544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:51.751264 2544 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca4f50f1d814974a64f388c1dd868a8 in term 1.
I20260929 10:25:51.751590 2545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca4f50f1d814974a64f388c1dd868a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8968cc1e0ecc47f697dd833587033cd4" is_pre_election: true
I20260929 10:25:51.751735 2545 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate bca4f50f1d814974a64f388c1dd868a8 in term 1. Re-sending same reply.
I20260929 10:25:51.752301 2620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bca4f50f1d814974a64f388c1dd868a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6827007bbe0446849dd7257a2dfc3cf8"
I20260929 10:25:51.752455 2620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:51.753882 2620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bca4f50f1d814974a64f388c1dd868a8 in term 1.
I20260929 10:25:51.754522 2669 leader_election.cc:302] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6827007bbe0446849dd7257a2dfc3cf8, bca4f50f1d814974a64f388c1dd868a8; no voters:
I20260929 10:25:51.756372 2764 leader_election.cc:288] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [CANDIDATE]: Term 1 election: Requested vote from peers 8968cc1e0ecc47f697dd833587033cd4 (127.21.33.190:34769), 6827007bbe0446849dd7257a2dfc3cf8 (127.21.33.189:37705)
I20260929 10:25:51.756549 2764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:51.756649 2764 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [term 1 LEADER]: Becoming Leader. State: Replica: bca4f50f1d814974a64f388c1dd868a8, State: Running, Role: LEADER
I20260929 10:25:51.757031 2764 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } }
I20260929 10:25:51.759028 2764 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca4f50f1d814974a64f388c1dd868a8. Latest consensus state: current_term: 1 leader_uuid: "bca4f50f1d814974a64f388c1dd868a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.759148 2764 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:51.759598 2779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:25:51.761122 2779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:25:51.768577 2620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [term 1 FOLLOWER]: Refusing update from remote peer bca4f50f1d814974a64f388c1dd868a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:51.769362 2777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:51.776441 2743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca4f50f1d814974a64f388c1dd868a8. Latest consensus state: current_term: 1 leader_uuid: "bca4f50f1d814974a64f388c1dd868a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.776595 2743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:25:51.778509 2545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [term 1 FOLLOWER]: Refusing update from remote peer bca4f50f1d814974a64f388c1dd868a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:51.779183 2777 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:51.783885 2730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca4f50f1d814974a64f388c1dd868a8. Latest consensus state: current_term: 1 leader_uuid: "bca4f50f1d814974a64f388c1dd868a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.784032 2730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:25:51.790519 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca4f50f1d814974a64f388c1dd868a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.790679 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:51.794751 2730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca4f50f1d814974a64f388c1dd868a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.794891 2730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:25:51.795475 2780 mvcc.cc:205] Tried to move back new op lower bound from 7334615252035727360 to 7334615252000444416. Current Snapshot: MvccSnapshot[applied={T|T < 7334615252035727360}]
I20260929 10:25:51.796821 2783 mvcc.cc:205] Tried to move back new op lower bound from 7334615252035727360 to 7334615252000444416. Current Snapshot: MvccSnapshot[applied={T|T < 7334615252035727360}]
I20260929 10:25:51.798135 2764 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca4f50f1d814974a64f388c1dd868a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.798257 2764 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:51.799597 2743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca4f50f1d814974a64f388c1dd868a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8968cc1e0ecc47f697dd833587033cd4" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 34769 } } peers { permanent_uuid: "6827007bbe0446849dd7257a2dfc3cf8" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 37705 } } peers { permanent_uuid: "bca4f50f1d814974a64f388c1dd868a8" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 44641 } } }
I20260929 10:25:51.799839 2743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:25:51.800340 2781 mvcc.cc:205] Tried to move back new op lower bound from 7334615252035727360 to 7334615252000444416. Current Snapshot: MvccSnapshot[applied={T|T < 7334615252035727360}]
I20260929 10:25:51.804154 2779 catalog_manager.cc:1347] Generated new cluster ID: f2ff5a77b54a442bab4e9d282b1d7aaf
I20260929 10:25:51.804224 2779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:25:51.820796 2779 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:25:51.822074 2779 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:25:51.843649 2779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca4f50f1d814974a64f388c1dd868a8: Generated new TSK 0
I20260929 10:25:51.844460 2779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:25:52.612056 2741 catalog_manager.cc:1259] Loaded cluster ID: f2ff5a77b54a442bab4e9d282b1d7aaf
I20260929 10:25:52.612159 2741 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: loading cluster ID for follower catalog manager: success
I20260929 10:25:52.614436 2741 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: acquiring CA information for follower catalog manager: success
I20260929 10:25:52.615717 2741 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8968cc1e0ecc47f697dd833587033cd4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:25:52.627342 2754 catalog_manager.cc:1259] Loaded cluster ID: f2ff5a77b54a442bab4e9d282b1d7aaf
I20260929 10:25:52.627413 2754 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: loading cluster ID for follower catalog manager: success
I20260929 10:25:52.628989 2754 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: acquiring CA information for follower catalog manager: success
I20260929 10:25:52.630110 2754 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6827007bbe0446849dd7257a2dfc3cf8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:25:52.902930 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 2503
I20260929 10:25:52.909540 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 2578
I20260929 10:25:52.915266 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 2653
2026-09-29T10:25:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3164 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:25:52 dist-test-slave-2rbl krb5kdc[2794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:25:52 dist-test-slave-2rbl krb5kdc[2794](info): set up 2 sockets
Sep 29 10:25:52 dist-test-slave-2rbl krb5kdc[2794](info): commencing operation
krb5kdc: starting...
W20260929 10:25:53.949972 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:25:54 dist-test-slave-2rbl krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:25:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:25:54.072921 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33837
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:54.183658 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:54.183948 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:54.184023 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:54.187994 2807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:54.188079 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:54.188118 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:54.188155 2807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:54.188189 2807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:54.193367 2807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:33837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:54.194377 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:54.195369 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:54.203972 2813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:54.204188 2807 server_base.cc:1034] running on GCE node
W20260929 10:25:54.203974 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:54.203974 2815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:54.204912 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:54.206018 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:54.207197 2807 hybrid_clock.cc:648] HybridClock initialized: now 1790677554207165 us; error 44 us; skew 500 ppm
Sep 29 10:25:54 dist-test-slave-2rbl krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677554, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:54.210134 2807 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:25:54.211045 2807 webserver.cc:460] Webserver started at http://127.21.33.190:34687/ using document root <none> and password file <none>
I20260929 10:25:54.211617 2807 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:54.211705 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:54.211927 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:54.214039 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "934918c2dc4f4c2d9fc12d3a9202226a"
format_stamp: "Formatted at 2026-09-29 10:25:54 on dist-test-slave-2rbl"
server_key: "ed12a00b4b85e1fbdb34562867974916"
server_key_iv: "86d2b2e43d04a4bfe3d1181401240b16"
server_key_version: "encryptionkey@0"
I20260929 10:25:54.214592 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "934918c2dc4f4c2d9fc12d3a9202226a"
format_stamp: "Formatted at 2026-09-29 10:25:54 on dist-test-slave-2rbl"
server_key: "ed12a00b4b85e1fbdb34562867974916"
server_key_iv: "86d2b2e43d04a4bfe3d1181401240b16"
server_key_version: "encryptionkey@0"
I20260929 10:25:54.218214 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:25:54.220539 2822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:54.221802 2807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:25:54.221935 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "934918c2dc4f4c2d9fc12d3a9202226a"
format_stamp: "Formatted at 2026-09-29 10:25:54 on dist-test-slave-2rbl"
server_key: "ed12a00b4b85e1fbdb34562867974916"
server_key_iv: "86d2b2e43d04a4bfe3d1181401240b16"
server_key_version: "encryptionkey@0"
I20260929 10:25:54.222040 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:54.250617 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:54.253733 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:54.253947 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:54.261334 2807 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:33837
I20260929 10:25:54.261374 2874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:33837 every 8 connection(s)
I20260929 10:25:54.262317 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:25:54.265115 2875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:54.269375 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 2807
I20260929 10:25:54.269495 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:25:54.269763 21638 external_mini_cluster.cc:1428] Setting key c7388a2161afcbd1f11e7c024dbd633c
I20260929 10:25:54.271139 2875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a: Bootstrap starting.
I20260929 10:25:54.273711 2875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:54.274585 2875 log.cc:824] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:54.276451 2875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a: No bootstrap required, opened a new log
Sep 29 10:25:54 dist-test-slave-2rbl krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:54.280346 2875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33837 } }
I20260929 10:25:54.280548 2875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:54.280628 2875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934918c2dc4f4c2d9fc12d3a9202226a, State: Initialized, Role: FOLLOWER
I20260929 10:25:54.281065 2875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33837 } }
I20260929 10:25:54.281185 2875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:25:54.281309 2875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:25:54.281436 2875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:54.282400 2875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33837 } }
I20260929 10:25:54.282761 2875 leader_election.cc:302] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 934918c2dc4f4c2d9fc12d3a9202226a; no voters:
I20260929 10:25:54.283066 2875 leader_election.cc:288] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:25:54.283165 2880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:54.283387 2880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [term 1 LEADER]: Becoming Leader. State: Replica: 934918c2dc4f4c2d9fc12d3a9202226a, State: Running, Role: LEADER
I20260929 10:25:54.283795 2880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33837 } }
I20260929 10:25:54.284214 2875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:54.285105 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 934918c2dc4f4c2d9fc12d3a9202226a. Latest consensus state: current_term: 1 leader_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33837 } } }
I20260929 10:25:54.285295 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:54.285588 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:54.271019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.21.33.190:33837)
0929 10:25:54.271414 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:54.271432 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:54.271482 (+ 50us) server_negotiation.cc:407] Connection header received
0929 10:25:54.272274 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:54.272299 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:54.272707 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:54.273041 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:54.273808 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:54.275183 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:54.276027 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:54.276687 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:54.276848 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:54.279712 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:54.279717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:54.279730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:54.279760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:54.281842 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:54.282310 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:54.282314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:54.282318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:54.282422 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:54.282745 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:54.282747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:54.282748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:54.283326 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:54.284146 (+ 820us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:54.284288 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 10:25:54.284568 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260929 10:25:54.285568 2881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "934918c2dc4f4c2d9fc12d3a9202226a" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33837 } } }
I20260929 10:25:54.285786 2881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:54.285861 2889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:25:54.288683 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:25:54.290050 2889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:25:54.295574 2889 catalog_manager.cc:1347] Generated new cluster ID: 4b5184c97e344360a78cad38e7e3c533
I20260929 10:25:54.295650 2889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:25:54.315688 2889 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:25:54.316586 2889 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:25:54.325057 2889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 934918c2dc4f4c2d9fc12d3a9202226a: Generated new TSK 0
I20260929 10:25:54.325757 2889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:25:54.344331 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 2807
2026-09-29T10:25:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1426 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:25:54 dist-test-slave-2rbl krb5kdc[2902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:25:54 dist-test-slave-2rbl krb5kdc[2902](info): set up 2 sockets
Sep 29 10:25:54 dist-test-slave-2rbl krb5kdc[2902](info): commencing operation
krb5kdc: starting...
W20260929 10:25:55.390051 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:25:55 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:25:55.536355 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35163
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:55.645381 2915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:55.645684 2915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:55.645754 2915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:55.649747 2915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:55.649828 2915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:55.649868 2915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:55.649904 2915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:55.649988 2915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:55.655143 2915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:55.656170 2915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:55.657235 2915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:55.665477 2920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:55.665588 2921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:55.665575 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:55.665663 2915 server_base.cc:1034] running on GCE node
I20260929 10:25:55.666249 2915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:55.667239 2915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:55.668399 2915 hybrid_clock.cc:648] HybridClock initialized: now 1790677555668380 us; error 30 us; skew 500 ppm
Sep 29 10:25:55 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677555, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:55.671087 2915 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:25:55.671972 2915 webserver.cc:460] Webserver started at http://127.21.33.190:40025/ using document root <none> and password file <none>
I20260929 10:25:55.672587 2915 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:55.672678 2915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:55.672950 2915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:55.674965 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "6b7f9269fa8446e1b7376d9da4d90238"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "1097026da817641a5a37980558827993"
server_key_iv: "3c77b2bcb145a43cba59ddd46ea19b04"
server_key_version: "encryptionkey@0"
I20260929 10:25:55.675534 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "6b7f9269fa8446e1b7376d9da4d90238"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "1097026da817641a5a37980558827993"
server_key_iv: "3c77b2bcb145a43cba59ddd46ea19b04"
server_key_version: "encryptionkey@0"
I20260929 10:25:55.679261 2915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:25:55.681658 2930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:55.682950 2915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:25:55.683074 2915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "6b7f9269fa8446e1b7376d9da4d90238"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "1097026da817641a5a37980558827993"
server_key_iv: "3c77b2bcb145a43cba59ddd46ea19b04"
server_key_version: "encryptionkey@0"
I20260929 10:25:55.683174 2915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:55.694983 2915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:55.697832 2915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:55.698027 2915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:55.704706 2915 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:35163
I20260929 10:25:55.704993 2982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:35163 every 8 connection(s)
I20260929 10:25:55.705768 2915 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:25:55.708631 2983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:55.712353 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 2915
I20260929 10:25:55.712471 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:25:55.712749 21638 external_mini_cluster.cc:1428] Setting key 3abd2847823d4e30701db22f72a853b9
I20260929 10:25:55.714949 2983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Bootstrap starting.
I20260929 10:25:55.717154 2983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:55.717947 2983 log.cc:824] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:55.719812 2983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: No bootstrap required, opened a new log
Sep 29 10:25:55 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:55.724078 2983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:55.724303 2983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:55.724392 2983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7f9269fa8446e1b7376d9da4d90238, State: Initialized, Role: FOLLOWER
I20260929 10:25:55.724910 2983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:55.725096 2983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:25:55.725171 2983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:25:55.725317 2983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:55.726461 2983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:55.726809 2983 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7f9269fa8446e1b7376d9da4d90238; no voters:
I20260929 10:25:55.727120 2983 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:25:55.727227 2988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:55.727442 2988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 LEADER]: Becoming Leader. State: Replica: 6b7f9269fa8446e1b7376d9da4d90238, State: Running, Role: LEADER
I20260929 10:25:55.727474 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:55.714303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.21.33.190:35163)
0929 10:25:55.714760 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:55.714780 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:55.714817 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:25:55.715529 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:55.715546 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:55.715877 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:55.716226 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:55.717044 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:55.718158 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:55.719063 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:55.719539 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:55.719706 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:55.722510 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:55.722516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:55.722530 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:55.722557 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:55.724241 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:55.724776 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:55.724782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:55.724792 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:55.724903 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:55.725247 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:55.725253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:55.725254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:55.725711 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:55.725862 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:55.726179 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 10:25:55.726485 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":142,"threads_started":1}
I20260929 10:25:55.728186 2988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:55.728449 2983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:55.730494 2990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7f9269fa8446e1b7376d9da4d90238. Latest consensus state: current_term: 1 leader_uuid: "6b7f9269fa8446e1b7376d9da4d90238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } } }
I20260929 10:25:55.730623 2990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:55.730911 2989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7f9269fa8446e1b7376d9da4d90238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } } }
I20260929 10:25:55.731288 2989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: This master's current role is: LEADER
W20260929 10:25:55.733563 3002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:25:55.733630 3002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:25:55.733704 3003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:25:55.734382 3003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:25:55.739967 3003 catalog_manager.cc:1347] Generated new cluster ID: 64b4e814742d4a24b25677814443ba56
I20260929 10:25:55.740044 3003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:25:55.750389 3003 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:25:55.751727 3003 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:25:55.763137 3003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Generated new TSK 0
I20260929 10:25:55.763996 3003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:25:55.796803 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35163
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:55.910115 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:55.910398 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:55.910468 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:55.914499 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:55.914562 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:55.914664 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:25:55.919924 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:55.920990 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:55.922058 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:55.932178 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:55.932209 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:55.932183 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:55.932354 3009 server_base.cc:1034] running on GCE node
I20260929 10:25:55.932958 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:55.933655 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:55.934813 3009 hybrid_clock.cc:648] HybridClock initialized: now 1790677555934779 us; error 45 us; skew 500 ppm
Sep 29 10:25:55 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677555, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:55.937541 3009 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:25:55.938373 3009 webserver.cc:460] Webserver started at http://127.21.33.129:34951/ using document root <none> and password file <none>
I20260929 10:25:55.938932 3009 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:55.938992 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:55.939193 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:55.941253 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "4fca613ef1bd41beb22e533afc421d45"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "61d1653ddad7446a926f4fc7bfa2466d"
server_key_iv: "9b7cb7e4093be191e7aaa50e76df9596"
server_key_version: "encryptionkey@0"
I20260929 10:25:55.941840 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "4fca613ef1bd41beb22e533afc421d45"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "61d1653ddad7446a926f4fc7bfa2466d"
server_key_iv: "9b7cb7e4093be191e7aaa50e76df9596"
server_key_version: "encryptionkey@0"
I20260929 10:25:55.945540 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:25:55.947932 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:55.949154 3009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:55.949319 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "4fca613ef1bd41beb22e533afc421d45"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "61d1653ddad7446a926f4fc7bfa2466d"
server_key_iv: "9b7cb7e4093be191e7aaa50e76df9596"
server_key_version: "encryptionkey@0"
I20260929 10:25:55.949438 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:55.980387 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:55.983832 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:55.984074 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:55.984710 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:55.985639 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:55.985713 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:55.985785 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:55.985824 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:55.994746 3009 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:44797
I20260929 10:25:55.994771 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:44797 every 8 connection(s)
I20260929 10:25:55.995626 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:25:56.003640 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3009
I20260929 10:25:56.003757 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:25:56.004032 21638 external_mini_cluster.cc:1428] Setting key 4bfb4f17f0fd6e40b84565ed95886c47
Sep 29 10:25:56 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677555, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:56.009966 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:55.997421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:41407 (local address 127.21.33.190:35163)
0929 10:25:55.997552 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:55.997555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:55.998409 (+ 854us) server_negotiation.cc:407] Connection header received
0929 10:25:55.999799 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:55.999802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:55.999851 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:55.999926 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:56.001091 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.001660 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.002756 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.003017 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.003132 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.005818 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:56.005825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:56.005831 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:56.005863 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:56.007771 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.008350 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.008354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.008356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.008445 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.008790 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.008794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.008795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.008981 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:56.009085 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.009633 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.009764 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:25:56.010730 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:55.997795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:41407)
0929 10:25:55.998251 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:55.998294 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:55.999588 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.000109 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.000116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.000472 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:56.000938 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.000950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.001758 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.001761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.002647 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.002655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.003175 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.003178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.003316 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.004063 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:56.004078 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:56.005649 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:56.007901 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:56.007907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:56.007922 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:56.008240 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:56.008540 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:56.008544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:56.008546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:56.008687 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:56.009082 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:56.009089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:56.009395 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.009578 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.009911 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
I20260929 10:25:56.012100 3138 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35163
I20260929 10:25:56.012339 3138 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:56.012823 3138 heartbeater.cc:511] Master 127.21.33.190:35163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:25:56.014528 2947 ts_manager.cc:194] Registered new tserver with Master: 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797)
I20260929 10:25:56.015726 2947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:41407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:25:56.033097 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35163
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:56.143105 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:56.143339 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:56.143409 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:56.147390 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:56.147444 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:56.147534 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:25:56.152693 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:56.153772 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:56.154778 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:25:56.164233 3143 server_base.cc:1034] running on GCE node
W20260929 10:25:56.164167 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:56.164152 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:56.164141 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:56.164819 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:56.165524 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:56.166687 3143 hybrid_clock.cc:648] HybridClock initialized: now 1790677556166653 us; error 47 us; skew 500 ppm
Sep 29 10:25:56 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677556, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:56.169338 3143 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:25:56.170181 3143 webserver.cc:460] Webserver started at http://127.21.33.130:39917/ using document root <none> and password file <none>
I20260929 10:25:56.170696 3143 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:56.170753 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:56.170912 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:56.172848 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "2c74fd95846e4dec9a27d2c8db81d787"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "3d652ccd1f93f6e3bf2f7f17b83074aa"
server_key_iv: "1029818b090e1836f83a3c576099d562"
server_key_version: "encryptionkey@0"
I20260929 10:25:56.173429 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "2c74fd95846e4dec9a27d2c8db81d787"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "3d652ccd1f93f6e3bf2f7f17b83074aa"
server_key_iv: "1029818b090e1836f83a3c576099d562"
server_key_version: "encryptionkey@0"
I20260929 10:25:56.177006 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:25:56.179421 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:56.180615 3143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:56.180755 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "2c74fd95846e4dec9a27d2c8db81d787"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "3d652ccd1f93f6e3bf2f7f17b83074aa"
server_key_iv: "1029818b090e1836f83a3c576099d562"
server_key_version: "encryptionkey@0"
I20260929 10:25:56.180862 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:56.193751 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:56.196539 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:56.196748 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:56.197405 3143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:56.198287 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:56.198343 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:56.198419 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:56.198455 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:56.207192 3143 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:33799
I20260929 10:25:56.207216 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:33799 every 8 connection(s)
I20260929 10:25:56.208062 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:25:56.208976 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3143
I20260929 10:25:56.209100 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:25:56.209429 21638 external_mini_cluster.cc:1428] Setting key 174f06e735b9dcc99505553d921a5e80
Sep 29 10:25:56 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677556, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:25:56.223016 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.210071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:40803 (local address 127.21.33.190:35163)
0929 10:25:56.210214 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.210219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.211128 (+ 909us) server_negotiation.cc:407] Connection header received
0929 10:25:56.213170 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.213174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.213252 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.213332 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:56.214512 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.215044 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.216178 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.216414 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.216571 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.219168 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:56.219172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:56.219175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:56.219203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:56.220760 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.221420 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.221424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.221426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.221539 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.221870 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.221875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.221876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.222074 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:56.222181 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.222698 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.222828 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:25:56.223825 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.210421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:40803)
0929 10:25:56.210961 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.211020 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:56.212944 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.213511 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.213518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.213876 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:56.214368 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.214381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.215177 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.215181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.216056 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.216065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.216608 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.216612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.216789 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.217412 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:56.217428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:56.218996 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:56.220925 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:56.220931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:56.220951 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:56.221302 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:56.221621 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:56.221626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:56.221628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:56.221775 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:56.222185 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:56.222194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:56.222448 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.222641 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.222973 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":133,"threads_started":1}
I20260929 10:25:56.225085 3272 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35163
I20260929 10:25:56.225358 3272 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:56.225843 3272 heartbeater.cc:511] Master 127.21.33.190:35163 requested a full tablet report, sending...
I20260929 10:25:56.226976 2947 ts_manager.cc:194] Registered new tserver with Master: 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
I20260929 10:25:56.227545 2947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:40803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:25:56.237455 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35163
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:56.348130 3277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:56.348412 3277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:56.348464 3277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:56.352592 3277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:56.352691 3277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:56.352809 3277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:25:56.358245 3277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:56.359287 3277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:56.360306 3277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:56.370472 3285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:56.370654 3283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:56.370546 3277 server_base.cc:1034] running on GCE node
W20260929 10:25:56.370570 3282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:56.371222 3277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:56.371873 3277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:56.373064 3277 hybrid_clock.cc:648] HybridClock initialized: now 1790677556373042 us; error 34 us; skew 500 ppm
Sep 29 10:25:56 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677556, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:56.375983 3277 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:25:56.376816 3277 webserver.cc:460] Webserver started at http://127.21.33.131:44845/ using document root <none> and password file <none>
I20260929 10:25:56.377420 3277 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:56.377493 3277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:56.377671 3277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:25:56.379735 3277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "8ad6c5f036094c74aca94261faf54512"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "7f16c990939d17b69ec92d59766275f8"
server_key_iv: "f7defed540557d47437b6774547c989e"
server_key_version: "encryptionkey@0"
I20260929 10:25:56.380280 3277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "8ad6c5f036094c74aca94261faf54512"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "7f16c990939d17b69ec92d59766275f8"
server_key_iv: "f7defed540557d47437b6774547c989e"
server_key_version: "encryptionkey@0"
I20260929 10:25:56.384100 3277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 10:25:56.386722 3292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:56.388149 3277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:25:56.388288 3277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "8ad6c5f036094c74aca94261faf54512"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "7f16c990939d17b69ec92d59766275f8"
server_key_iv: "f7defed540557d47437b6774547c989e"
server_key_version: "encryptionkey@0"
I20260929 10:25:56.388408 3277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:56.413043 3277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:56.416656 3277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:56.416874 3277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:56.417559 3277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:56.418409 3277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:56.418486 3277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:56.418558 3277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:56.418607 3277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:56.427292 3277 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:43981
I20260929 10:25:56.427328 3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:43981 every 8 connection(s)
I20260929 10:25:56.428268 3277 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:25:56.433797 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3277
I20260929 10:25:56.433894 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:25:56.434186 21638 external_mini_cluster.cc:1428] Setting key 553ce3bab9b73d9cb4e307735c485fd2
Sep 29 10:25:56 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677556, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:56.442303 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.430307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:39931 (local address 127.21.33.190:35163)
0929 10:25:56.430501 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.430510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.431308 (+ 798us) server_negotiation.cc:407] Connection header received
0929 10:25:56.432694 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.432697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.432753 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.432827 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:56.434075 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.434710 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.436356 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.436577 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.436659 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.439012 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:56.439015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:56.439017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:56.439041 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:56.440280 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.440757 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.440760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.440761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.440838 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.441279 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.441282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.441283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.441463 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:56.441554 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.442077 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.442186 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:25:56.443070 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.430704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:39931)
0929 10:25:56.431167 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.431210 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:56.432485 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.432991 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.432997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.433403 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:56.433904 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.433928 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.434856 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.434859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.436219 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.436230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.436702 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.436705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.436849 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.437441 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:56.437456 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:56.438869 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:56.440397 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:56.440402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:56.440412 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:56.440667 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:56.440955 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:56.440958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:56.440960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:56.441124 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:56.441599 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:56.441604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:56.441830 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.442016 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.442272 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260929 10:25:56.444140 3406 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35163
I20260929 10:25:56.444377 3406 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:56.444842 3406 heartbeater.cc:511] Master 127.21.33.190:35163 requested a full tablet report, sending...
I20260929 10:25:56.445914 2947 ts_manager.cc:194] Registered new tserver with Master: 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981)
I20260929 10:25:56.446501 2947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:39931
I20260929 10:25:56.448104 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:25:56.457381 2986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.450065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.21.33.190:35163)
0929 10:25:56.450234 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.450238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.450375 (+ 137us) server_negotiation.cc:407] Connection header received
0929 10:25:56.450533 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.450537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.450578 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.450662 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:56.451395 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.451912 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.452753 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.453009 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.453112 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.454532 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:56.454536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:56.454539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:56.454568 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:56.455722 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.456136 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.456139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.456141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.456217 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:56.456486 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:56.456489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:56.456491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:56.456703 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:56.456839 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.457062 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.457224 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 10:25:56.460083 2947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:25:56.463183 2947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:25:56.477964 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.472426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.21.33.130:33799)
0929 10:25:56.472751 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.472762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.473138 (+ 376us) server_negotiation.cc:407] Connection header received
0929 10:25:56.473274 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.473279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.473458 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.473589 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:56.473792 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.474637 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.476067 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.476891 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.476972 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.477111 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.477690 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.477800 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260929 10:25:56.477985 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.471611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44797 (local address 127.0.0.1:45172)
0929 10:25:56.471954 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.472000 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:56.472121 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.473076 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.473085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.473112 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:56.473295 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.473304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.474491 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.474494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.475931 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.475937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.477244 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.477248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.477485 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.477527 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.477643 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.477807 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":126,"spinlock_wait_cycles":1792,"thread_start_us":106,"threads_started":1}
I20260929 10:25:56.477985 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.471685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33799 (local address 127.0.0.1:37146)
0929 10:25:56.473038 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.473051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:56.473136 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.473594 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.473597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.473615 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:56.473701 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.473706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.474787 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.474790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.475923 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.475934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.477244 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.477248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.477480 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.477526 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.477643 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.477807 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":87,"thread_start_us":100,"threads_started":1}
I20260929 10:25:56.478472 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.471704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.21.33.129:44797)
0929 10:25:56.472346 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.472363 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.472387 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:25:56.472467 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.472472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.472707 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.472866 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:56.473438 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.474374 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.476073 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.476891 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.476972 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.477111 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.478206 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.478306 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":78,"threads_started":1}
I20260929 10:25:56.478566 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.472361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:43981 (local address 127.0.0.1:40194)
0929 10:25:56.474586 (+ 2225us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.474597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:56.474696 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.475083 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.475085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.475103 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:56.475196 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.475203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.476411 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.476416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.477150 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.477158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.478182 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.478186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.478288 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.478312 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.478364 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.478432 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2172,"thread_start_us":93,"threads_started":1}
I20260929 10:25:56.479198 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.472546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.21.33.131:43981)
0929 10:25:56.472913 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.472923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.474685 (+ 1762us) server_negotiation.cc:407] Connection header received
0929 10:25:56.474801 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.474805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.474942 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.475092 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:56.475298 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.476291 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.477304 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.477992 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.478796 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.478931 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.478986 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.479067 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":149,"threads_started":1}
I20260929 10:25:56.481448 3072 tablet_service.cc:1467] Processing CreateTablet for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 (DEFAULT_TABLE table=test-table [id=9acab87f9e184e76b7922b7fad47d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:56.481448 3206 tablet_service.cc:1467] Processing CreateTablet for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 (DEFAULT_TABLE table=test-table [id=9acab87f9e184e76b7922b7fad47d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:56.482601 3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:56.482601 3072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:56.485792 3340 tablet_service.cc:1467] Processing CreateTablet for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 (DEFAULT_TABLE table=test-table [id=9acab87f9e184e76b7922b7fad47d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:56.486740 3340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:56.490212 3423 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Bootstrap starting.
I20260929 10:25:56.491494 3424 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Bootstrap starting.
I20260929 10:25:56.493268 3423 tablet_bootstrap.cc:654] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:56.493426 3424 tablet_bootstrap.cc:654] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:56.493867 3425 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Bootstrap starting.
I20260929 10:25:56.494743 3424 log.cc:824] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:56.495007 3423 log.cc:824] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:56.496249 3425 tablet_bootstrap.cc:654] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:56.496866 3424 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: No bootstrap required, opened a new log
I20260929 10:25:56.497008 3423 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: No bootstrap required, opened a new log
I20260929 10:25:56.497109 3424 ts_tablet_manager.cc:1397] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:25:56.497220 3423 ts_tablet_manager.cc:1397] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:25:56.497598 3425 log.cc:824] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:56.499524 3425 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: No bootstrap required, opened a new log
I20260929 10:25:56.499703 3425 ts_tablet_manager.cc:1397] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:25:56.500447 3424 raft_consensus.cc:348] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.500687 3424 raft_consensus.cc:374] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:56.500741 3424 raft_consensus.cc:729] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fca613ef1bd41beb22e533afc421d45, State: Initialized, Role: FOLLOWER
I20260929 10:25:56.500882 3423 raft_consensus.cc:348] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.501121 3423 raft_consensus.cc:374] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:56.501166 3423 raft_consensus.cc:729] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c74fd95846e4dec9a27d2c8db81d787, State: Initialized, Role: FOLLOWER
I20260929 10:25:56.501272 3424 consensus_queue.cc:260] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.501677 3423 consensus_queue.cc:260] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.502298 3424 ts_tablet_manager.cc:1428] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:25:56.502578 3423 ts_tablet_manager.cc:1428] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:25:56.502674 3138 heartbeater.cc:503] Master 127.21.33.190:35163 was elected leader, sending a full tablet report...
I20260929 10:25:56.502885 3272 heartbeater.cc:503] Master 127.21.33.190:35163 was elected leader, sending a full tablet report...
I20260929 10:25:56.503432 3425 raft_consensus.cc:348] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.503680 3425 raft_consensus.cc:374] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:56.503739 3425 raft_consensus.cc:729] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad6c5f036094c74aca94261faf54512, State: Initialized, Role: FOLLOWER
I20260929 10:25:56.504176 3425 consensus_queue.cc:260] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.504830 3406 heartbeater.cc:503] Master 127.21.33.190:35163 was elected leader, sending a full tablet report...
I20260929 10:25:56.505038 3425 ts_tablet_manager.cc:1428] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:25:56.622787 3431 raft_consensus.cc:482] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:25:56.623037 3431 raft_consensus.cc:504] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.624298 3431 leader_election.cc:288] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797), 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
I20260929 10:25:56.628247 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.624515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44797 (local address 127.21.33.131:60769)
0929 10:25:56.624701 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.624717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:56.624812 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.625110 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.625115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.625135 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:56.625281 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.625291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.626162 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.626165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.627045 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.627055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.627789 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.627792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.627901 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.627927 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.628049 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.628120 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":1}
I20260929 10:25:56.628268 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.624651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:60769 (local address 127.21.33.129:44797)
0929 10:25:56.624782 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.624788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.624808 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:25:56.624937 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.624940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.625000 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.625083 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:56.625417 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.626043 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.627174 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.627677 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.627771 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.627810 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.628064 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.628139 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:25:56.628531 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.624665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:46653 (local address 127.21.33.130:33799)
0929 10:25:56.624800 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.624805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.625089 (+ 284us) server_negotiation.cc:407] Connection header received
0929 10:25:56.625175 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.625178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.625261 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.625350 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:56.625720 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.626283 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.627466 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.627960 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.628074 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.628119 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.628310 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.628388 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:25:56.628551 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.624546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33799 (local address 127.21.33.131:46653)
0929 10:25:56.624963 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.624989 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:56.625093 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:56.625416 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:56.625420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:56.625447 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:56.625572 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.625579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.626412 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.626415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.627344 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:56.627354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.628076 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.628079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:56.628183 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.628204 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:25:56.628282 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:25:56.628354 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":25,"thread_start_us":125,"threads_started":1}
I20260929 10:25:56.628970 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24b2c4ad2f4b5eba7fc55adae08e34" candidate_uuid: "8ad6c5f036094c74aca94261faf54512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fca613ef1bd41beb22e533afc421d45" is_pre_election: true
I20260929 10:25:56.629364 3092 raft_consensus.cc:2454] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad6c5f036094c74aca94261faf54512 in term 0.
I20260929 10:25:56.629325 3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24b2c4ad2f4b5eba7fc55adae08e34" candidate_uuid: "8ad6c5f036094c74aca94261faf54512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c74fd95846e4dec9a27d2c8db81d787" is_pre_election: true
I20260929 10:25:56.629628 3226 raft_consensus.cc:2454] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad6c5f036094c74aca94261faf54512 in term 0.
I20260929 10:25:56.629865 3296 leader_election.cc:302] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fca613ef1bd41beb22e533afc421d45, 8ad6c5f036094c74aca94261faf54512; no voters:
I20260929 10:25:56.630195 3431 raft_consensus.cc:2781] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:25:56.630300 3431 raft_consensus.cc:482] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:25:56.630337 3431 raft_consensus.cc:3037] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:56.631368 3431 raft_consensus.cc:504] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.631729 3431 leader_election.cc:288] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [CANDIDATE]: Term 1 election: Requested vote from peers 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797), 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
I20260929 10:25:56.632051 3226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24b2c4ad2f4b5eba7fc55adae08e34" candidate_uuid: "8ad6c5f036094c74aca94261faf54512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c74fd95846e4dec9a27d2c8db81d787"
I20260929 10:25:56.632066 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d24b2c4ad2f4b5eba7fc55adae08e34" candidate_uuid: "8ad6c5f036094c74aca94261faf54512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fca613ef1bd41beb22e533afc421d45"
I20260929 10:25:56.632157 3226 raft_consensus.cc:3037] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:56.632189 3092 raft_consensus.cc:3037] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:56.633145 3092 raft_consensus.cc:2454] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad6c5f036094c74aca94261faf54512 in term 1.
I20260929 10:25:56.633287 3226 raft_consensus.cc:2454] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad6c5f036094c74aca94261faf54512 in term 1.
I20260929 10:25:56.633452 3296 leader_election.cc:302] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fca613ef1bd41beb22e533afc421d45, 8ad6c5f036094c74aca94261faf54512; no voters:
I20260929 10:25:56.633680 3431 raft_consensus.cc:2781] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:56.633864 3431 raft_consensus.cc:686] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad6c5f036094c74aca94261faf54512, State: Running, Role: LEADER
I20260929 10:25:56.634171 3431 consensus_queue.cc:237] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:56.637439 2946 catalog_manager.cc:5495] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad6c5f036094c74aca94261faf54512 (127.21.33.131). New cstate: current_term: 1 leader_uuid: "8ad6c5f036094c74aca94261faf54512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } health_report { overall_health: HEALTHY } } }
I20260929 10:25:56.668336 3422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.665045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.21.33.131:43981)
0929 10:25:56.665163 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:56.665166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:56.665179 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:25:56.665334 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:56.665336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:56.665391 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:56.665482 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:25:56.665688 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.666194 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.667018 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:56.667220 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:56.667292 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:56.667548 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:25:56.667961 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:25:56.668079 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:56.668162 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 10:25:56.668210 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:25:56.674093 3226 raft_consensus.cc:1261] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 FOLLOWER]: Refusing update from remote peer 8ad6c5f036094c74aca94261faf54512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:56.674100 3092 raft_consensus.cc:1261] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Refusing update from remote peer 8ad6c5f036094c74aca94261faf54512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:56.674784 3431 consensus_queue.cc:1035] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:56.674960 3433 consensus_queue.cc:1035] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:25:56.679147 3407 tablet.cc:2369] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:25:56.685874 3437 mvcc.cc:205] Tried to move back new op lower bound from 7334615272130076672 to 7334615271976218624. Current Snapshot: MvccSnapshot[applied={T|T < 7334615272130076672 or (T in {7334615272130076672})}]
I20260929 10:25:56.690145 2947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:56.690302 2947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:56.693065 2947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Sending DeleteTablet for 3 replicas of tablet 7d24b2c4ad2f4b5eba7fc55adae08e34
I20260929 10:25:56.693938 3206 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:56 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:37146
I20260929 10:25:56.693938 3340 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:56 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:40194
I20260929 10:25:56.694494 3072 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:56 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:45172
I20260929 10:25:56.694522 3445 tablet_replica.cc:330] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: stopping tablet replica
I20260929 10:25:56.694900 3445 raft_consensus.cc:2229] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:56.694962 3447 tablet_replica.cc:330] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: stopping tablet replica
I20260929 10:25:56.695240 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 3009
I20260929 10:25:56.695252 3445 raft_consensus.cc:2258] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:56.695497 3447 raft_consensus.cc:2229] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:56.696935 3445 ts_tablet_manager.cc:1905] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:56.699891 3445 ts_tablet_manager.cc:1918] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:56.700004 3445 log.cc:1196] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/7d24b2c4ad2f4b5eba7fc55adae08e34
I20260929 10:25:56.700347 3445 ts_tablet_manager.cc:1939] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 2c74fd95846e4dec9a27d2c8db81d787: Deleting consensus metadata
I20260929 10:25:56.700311 3446 tablet_replica.cc:330] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: stopping tablet replica
I20260929 10:25:56.700793 3446 raft_consensus.cc:2229] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:25:56.701273 3446 raft_consensus.cc:2258] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:56.701781 2933 catalog_manager.cc:4841] TS 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799): tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 (table test-table [id=9acab87f9e184e76b7922b7fad47d028]) successfully deleted
W20260929 10:25:56.702481 2934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:44797 (error 108)
I20260929 10:25:56.703169 3446 ts_tablet_manager.cc:1905] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:56.703326 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 3143
I20260929 10:25:56.706496 3446 ts_tablet_manager.cc:1918] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:56.706609 3446 log.cc:1196] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/7d24b2c4ad2f4b5eba7fc55adae08e34
I20260929 10:25:56.706938 3446 ts_tablet_manager.cc:1939] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 8ad6c5f036094c74aca94261faf54512: Deleting consensus metadata
I20260929 10:25:56.708169 2934 catalog_manager.cc:4841] TS 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981): tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 (table test-table [id=9acab87f9e184e76b7922b7fad47d028]) successfully deleted
I20260929 10:25:56.708760 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:56.708467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44797 (local address 127.0.0.1:45176)
0929 10:25:56.708623 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:56.708689 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:44797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 10:25:56.709309 2934 catalog_manager.cc:4584] TS 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34: Network error: Client connection negotiation failed: client connection to 127.21.33.129:44797: connect: Connection refused (error 111)
I20260929 10:25:56.710101 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 3277
I20260929 10:25:56.716256 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 2915
I20260929 10:25:56.721549 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35163
--webserver_interface=127.21.33.190
--webserver_port=40025
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:56.829716 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:56.829994 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:56.830066 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:56.834010 3449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:56.834090 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:56.834129 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:56.834182 3449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:56.834223 3449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:56.839349 3449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=40025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:56.840582 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:56.841818 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:56.855412 3455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:56.855402 3458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:56.855449 3449 server_base.cc:1034] running on GCE node
W20260929 10:25:56.855402 3456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:56.856287 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:56.857491 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:56.858657 3449 hybrid_clock.cc:648] HybridClock initialized: now 1790677556858623 us; error 48 us; skew 500 ppm
Sep 29 10:25:56 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677556, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:56.861599 3449 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:25:56.862499 3449 webserver.cc:460] Webserver started at http://127.21.33.190:40025/ using document root <none> and password file <none>
I20260929 10:25:56.863076 3449 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:56.863178 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:56.866959 3449 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 10:25:56.869040 3465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:56.870239 3449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:56.870369 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "6b7f9269fa8446e1b7376d9da4d90238"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "1097026da817641a5a37980558827993"
server_key_iv: "3c77b2bcb145a43cba59ddd46ea19b04"
server_key_version: "encryptionkey@0"
I20260929 10:25:56.870958 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:56.900920 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:56.904392 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:56.904609 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:56.911640 3449 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:35163
I20260929 10:25:56.911922 3517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:35163 every 8 connection(s)
I20260929 10:25:56.912740 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:25:56.916903 3518 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:25:56.917785 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3449
I20260929 10:25:56.918280 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:44797
--local_ip_for_outbound_sockets=127.21.33.129
--tserver_master_addrs=127.21.33.190:35163
--webserver_port=34951
--webserver_interface=127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:56.918946 3518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Bootstrap starting.
I20260929 10:25:56.935865 3518 log.cc:824] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:56.944469 3518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:25:56.944868 3518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Bootstrap complete.
I20260929 10:25:56.949121 3518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:56.949574 3518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7f9269fa8446e1b7376d9da4d90238, State: Initialized, Role: FOLLOWER
I20260929 10:25:56.950129 3518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:56.950241 3518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:25:56.950327 3518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:25:56.950436 3518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:25:56.952301 3518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:56.952658 3518 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7f9269fa8446e1b7376d9da4d90238; no voters:
I20260929 10:25:56.952975 3518 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:25:56.953068 3522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:25:56.953322 3522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 LEADER]: Becoming Leader. State: Replica: 6b7f9269fa8446e1b7376d9da4d90238, State: Running, Role: LEADER
I20260929 10:25:56.953624 3522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:56.954249 3518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:56.954574 3522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7f9269fa8446e1b7376d9da4d90238. Latest consensus state: current_term: 2 leader_uuid: "6b7f9269fa8446e1b7376d9da4d90238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } } }
I20260929 10:25:56.954685 3522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:56.954949 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b7f9269fa8446e1b7376d9da4d90238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } } }
I20260929 10:25:56.955055 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:56.955120 3526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:25:56.959968 3526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:25:56.960811 3526 catalog_manager.cc:1259] Loaded cluster ID: 64b4e814742d4a24b25677814443ba56
I20260929 10:25:56.960875 3526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:25:56.962294 3526 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 10:25:56.967198 3526 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 10:25:56.972440 3526 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:25:56.973217 3526 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Loaded TSK: 0
I20260929 10:25:56.974390 3526 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:25:57.031931 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:57.032197 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:57.032267 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:57.036305 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:57.036381 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:57.036471 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:25:57.041559 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:44797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=34951
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:57.042541 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:57.043512 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:57.052904 3546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:57.052889 3545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:57.052882 3548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:57.053704 3520 server_base.cc:1034] running on GCE node
I20260929 10:25:57.054109 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:57.054790 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:57.055964 3520 hybrid_clock.cc:648] HybridClock initialized: now 1790677557055935 us; error 45 us; skew 500 ppm
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:57.058889 3520 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:25:57.059828 3520 webserver.cc:460] Webserver started at http://127.21.33.129:34951/ using document root <none> and password file <none>
I20260929 10:25:57.060420 3520 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:57.060539 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:57.064774 3520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:25:57.067060 3555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.068403 3520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:25:57.068557 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "4fca613ef1bd41beb22e533afc421d45"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "61d1653ddad7446a926f4fc7bfa2466d"
server_key_iv: "9b7cb7e4093be191e7aaa50e76df9596"
server_key_version: "encryptionkey@0"
I20260929 10:25:57.069109 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:57.091178 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:57.094249 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:57.094449 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:57.095054 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:57.096211 3562 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:25:57.101112 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:25:57.101186 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 10:25:57.101305 3520 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:25:57.102870 3520 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:25:57.102936 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 10:25:57.103201 3562 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Bootstrap starting.
I20260929 10:25:57.112974 3520 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:44797
I20260929 10:25:57.113047 3669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:44797 every 8 connection(s)
I20260929 10:25:57.113888 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:25:57.115023 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3520
I20260929 10:25:57.115576 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:33799
--local_ip_for_outbound_sockets=127.21.33.130
--tserver_master_addrs=127.21.33.190:35163
--webserver_port=39917
--webserver_interface=127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:57.123049 3562 log.cc:824] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:57.132014 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.115807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:57311)
0929 10:25:57.116296 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.116346 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.117437 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.118921 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.118928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.119284 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:57.119722 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.119736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.121716 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.121723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.123307 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.123324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.124074 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.124078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.124257 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.124857 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:57.124873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:57.126338 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:57.128866 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.128872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.128885 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.129249 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.129683 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.129686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.129688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.129857 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.130563 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:57.130570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:57.130870 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.131023 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.131323 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":132,"threads_started":1}
I20260929 10:25:57.132422 3562 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:25:57.132834 3562 tablet_bootstrap.cc:492] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Bootstrap complete.
I20260929 10:25:57.133721 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.115781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:57311 (local address 127.21.33.190:35163)
0929 10:25:57.117058 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.117078 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.117122 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:25:57.117965 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.117985 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.118477 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.118865 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:57.119893 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.121524 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.123457 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.123924 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.124120 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.126527 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:57.126532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:57.126543 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:57.126571 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:57.128715 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.129395 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.129400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.129406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.129563 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.129970 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.129973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.129975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.130429 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:57.132663 (+ 2234us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.132778 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.133046 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":104,"threads_started":1}
I20260929 10:25:57.134270 3562 ts_tablet_manager.cc:1397] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260929 10:25:57.135737 3670 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35163
I20260929 10:25:57.135972 3670 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:57.136441 3670 heartbeater.cc:511] Master 127.21.33.190:35163 requested a full tablet report, sending...
I20260929 10:25:57.137147 3562 raft_consensus.cc:348] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:57.137792 3562 raft_consensus.cc:729] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fca613ef1bd41beb22e533afc421d45, State: Initialized, Role: FOLLOWER
I20260929 10:25:57.138270 3562 consensus_queue.cc:260] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } }
I20260929 10:25:57.138911 3482 ts_manager.cc:194] Registered new tserver with Master: 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797)
I20260929 10:25:57.139267 3562 ts_tablet_manager.cc:1428] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:25:57.142230 3482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:57311
I20260929 10:25:57.144548 3670 heartbeater.cc:503] Master 127.21.33.190:35163 was elected leader, sending a full tablet report...
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677556, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:25:57.152195 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.141671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44797 (local address 127.0.0.1:45180)
0929 10:25:57.142068 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.142126 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.142252 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.143132 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.143137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.143572 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:57.143697 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.143705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.144906 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.144910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.145672 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.145681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.146380 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.146384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.146501 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.147026 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:57.147040 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:57.148553 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:57.150464 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.150473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.150478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.150712 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.151004 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.151008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.151010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.151096 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.151546 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:57.151552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:57.151681 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.151816 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.152009 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":124,"threads_started":1}
I20260929 10:25:57.152199 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.141732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.21.33.129:44797)
0929 10:25:57.142522 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.142533 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.142555 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:25:57.142621 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.142626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.142834 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.142965 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:57.143831 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.144804 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.145788 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.146244 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.146343 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.148704 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:57.148718 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:57.148728 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:57.148749 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:57.150336 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.150826 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.150832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.150838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.150913 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.151183 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.151188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.151190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.151418 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:57.151572 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.151896 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.152049 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":94,"threads_started":1}
I20260929 10:25:57.153172 3604 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:56 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:45180
I20260929 10:25:57.154206 3679 tablet_replica.cc:330] stopping tablet replica
I20260929 10:25:57.154487 3679 raft_consensus.cc:2229] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:57.154765 3679 raft_consensus.cc:2258] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:57.155839 3679 ts_tablet_manager.cc:1905] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:57.159475 3679 ts_tablet_manager.cc:1918] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:57.159554 3679 log.cc:1196] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/7d24b2c4ad2f4b5eba7fc55adae08e34
I20260929 10:25:57.159832 3679 ts_tablet_manager.cc:1939] T 7d24b2c4ad2f4b5eba7fc55adae08e34 P 4fca613ef1bd41beb22e533afc421d45: Deleting consensus metadata
I20260929 10:25:57.161405 3469 catalog_manager.cc:4841] TS 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797): tablet 7d24b2c4ad2f4b5eba7fc55adae08e34 (table test-table [id=9acab87f9e184e76b7922b7fad47d028]) successfully deleted
W20260929 10:25:57.239261 3674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:57.239565 3674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:57.239639 3674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:57.244155 3674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:57.244251 3674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:57.244349 3674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:25:57.250027 3674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:33799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=39917
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:57.251099 3674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:57.252207 3674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:57.262161 3687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:57.262257 3674 server_base.cc:1034] running on GCE node
W20260929 10:25:57.262163 3684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:57.262187 3685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:57.262823 3674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:57.263495 3674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:57.264668 3674 hybrid_clock.cc:648] HybridClock initialized: now 1790677557264651 us; error 30 us; skew 500 ppm
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:57.267370 3674 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:25:57.268201 3674 webserver.cc:460] Webserver started at http://127.21.33.130:39917/ using document root <none> and password file <none>
I20260929 10:25:57.268754 3674 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:57.268844 3674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:57.272565 3674 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:57.274735 3694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.276001 3674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 10:25:57.276127 3674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "2c74fd95846e4dec9a27d2c8db81d787"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "3d652ccd1f93f6e3bf2f7f17b83074aa"
server_key_iv: "1029818b090e1836f83a3c576099d562"
server_key_version: "encryptionkey@0"
I20260929 10:25:57.276638 3674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:57.290612 3674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:57.293324 3674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:57.293506 3674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:57.294097 3674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:57.294868 3674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:57.294914 3674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.294989 3674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:57.295027 3674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.303599 3674 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:33799
I20260929 10:25:57.303624 3807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:33799 every 8 connection(s)
I20260929 10:25:57.304646 3674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:57.314177 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3674
I20260929 10:25:57.314584 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:43981
--local_ip_for_outbound_sockets=127.21.33.131
--tserver_master_addrs=127.21.33.190:35163
--webserver_port=44845
--webserver_interface=127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:57.318007 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.306372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:59485 (local address 127.21.33.190:35163)
0929 10:25:57.306523 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.306527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.307310 (+ 783us) server_negotiation.cc:407] Connection header received
0929 10:25:57.308617 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.308621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.308666 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.308742 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:57.309928 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.310455 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.311556 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.311786 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.311873 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.314228 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:57.314231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:57.314233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:57.314276 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:57.315871 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.316330 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.316334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.316337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.316432 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.316744 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.316748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.316750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.316984 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:57.317093 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.317607 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.317762 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:25:57.318568 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.306751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:59485)
0929 10:25:57.307170 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.307209 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.308428 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.308895 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.308901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.309314 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:57.309786 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.309799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.310571 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.310573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.311423 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.311432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.311911 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.311914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.312049 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.312586 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:57.312601 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:57.314041 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:57.315982 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.315988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.316002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.316238 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.316531 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.316533 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.316534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.316654 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.317089 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:57.317094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:57.317361 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.317519 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.317750 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":110,"threads_started":1}
I20260929 10:25:57.319867 3808 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35163
I20260929 10:25:57.320113 3808 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:57.320567 3808 heartbeater.cc:511] Master 127.21.33.190:35163 requested a full tablet report, sending...
I20260929 10:25:57.321583 3482 ts_manager.cc:194] Registered new tserver with Master: 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
I20260929 10:25:57.322136 3482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:59485
W20260929 10:25:57.426254 3811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:57.426549 3811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:57.426620 3811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:57.430696 3811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:57.430923 3811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:57.431072 3811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:25:57.436627 3811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:43981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=44845
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:57.437760 3811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:57.438773 3811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:57.448148 3816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:57.448280 3819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:57.448232 3811 server_base.cc:1034] running on GCE node
W20260929 10:25:57.448143 3817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:57.448876 3811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:57.449585 3811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:57.450783 3811 hybrid_clock.cc:648] HybridClock initialized: now 1790677557450747 us; error 49 us; skew 500 ppm
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:57.453456 3811 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:25:57.454257 3811 webserver.cc:460] Webserver started at http://127.21.33.131:44845/ using document root <none> and password file <none>
I20260929 10:25:57.454859 3811 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:57.454960 3811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:57.459015 3811 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:25:57.461087 3826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.462247 3811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:57.462371 3811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "8ad6c5f036094c74aca94261faf54512"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "7f16c990939d17b69ec92d59766275f8"
server_key_iv: "f7defed540557d47437b6774547c989e"
server_key_version: "encryptionkey@0"
I20260929 10:25:57.462872 3811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:57.515856 3811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:57.518848 3811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:57.519050 3811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:57.519661 3811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:57.520483 3811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:57.520556 3811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.520627 3811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:57.520675 3811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.529271 3811 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:43981
I20260929 10:25:57.529289 3939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:43981 every 8 connection(s)
I20260929 10:25:57.530249 3811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:25:57.531862 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3811
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:25:57.541244 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.532523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.21.33.190:35163)
0929 10:25:57.533049 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.533052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.533063 (+ 11us) server_negotiation.cc:407] Connection header received
0929 10:25:57.533101 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.533103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.533179 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.533284 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:57.534033 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.534623 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.535810 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.536087 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.536196 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.537851 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:57.537854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:57.537856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:57.537877 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:57.539532 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.540014 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.540019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.540021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.540103 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.540377 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.540381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.540383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.540611 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:57.540819 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.540954 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.541064 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":87,"threads_started":1}
I20260929 10:25:57.544550 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.532016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:59553 (local address 127.21.33.190:35163)
0929 10:25:57.532163 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.532166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.533153 (+ 987us) server_negotiation.cc:407] Connection header received
0929 10:25:57.534528 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.534531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.534572 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.534661 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:57.536046 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.536741 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.538267 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.538484 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.538589 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.540983 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:57.540987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:57.540990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:57.541048 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:57.542644 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.543141 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.543146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.543148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.543226 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:57.543559 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:57.543564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:57.543566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:57.543728 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:57.543831 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.544292 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.544398 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
I20260929 10:25:57.545408 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.532520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:59553)
0929 10:25:57.532999 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.533050 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.534346 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.534834 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.534843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.535305 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:57.535925 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.535941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.536866 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.536869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.538145 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.538154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.538636 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.538640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.538806 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.539265 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:57.539280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:57.540819 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:57.542769 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.542774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.542788 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.543042 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.543322 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:57.543325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:57.543327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:57.543467 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:57.543830 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:57.543836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:57.544071 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.544253 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.544578 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260929 10:25:57.546468 3940 heartbeater.cc:347] Connected to a master server at 127.21.33.190:35163
I20260929 10:25:57.546684 3940 heartbeater.cc:464] Registering TS with master...
I20260929 10:25:57.547135 3940 heartbeater.cc:511] Master 127.21.33.190:35163 requested a full tablet report, sending...
I20260929 10:25:57.548125 3482 ts_manager.cc:194] Registered new tserver with Master: 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981)
I20260929 10:25:57.548656 3482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:59553
I20260929 10:25:57.554418 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:25:57.558350 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.554995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.21.33.190:35163)
0929 10:25:57.555098 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.555102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.555281 (+ 179us) server_negotiation.cc:407] Connection header received
0929 10:25:57.555487 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.555490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.555579 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.555662 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:25:57.555861 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.556335 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.557166 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.557410 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.557489 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.557749 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:25:57.558090 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:25:57.558173 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.558223 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.558262 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:25:57.559120 3482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:25:57.562288 3482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:25:57.570063 3604 tablet_service.cc:1467] Processing CreateTablet for tablet 715bdfd22c3e417aa1001efddb937a9e (DEFAULT_TABLE table=test-table [id=5ebe4e3604e84a73aaa40b81291e2cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:57.570715 3604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715bdfd22c3e417aa1001efddb937a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:57.573552 3562 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Bootstrap starting.
I20260929 10:25:57.574319 3562 tablet_bootstrap.cc:654] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:57.575233 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.569137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33799 (local address 127.0.0.1:37156)
0929 10:25:57.569412 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.569426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.569512 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.570446 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.570450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.570468 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:57.570571 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.570577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.571600 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.571605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.573244 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.573255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.574615 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.574620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.574853 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.574904 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.574994 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.575065 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":124,"thread_start_us":152,"threads_started":1}
I20260929 10:25:57.575289 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.569602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.21.33.130:33799)
0929 10:25:57.569880 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.569891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.569917 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:25:57.569978 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.569982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.570120 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.570278 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:57.570702 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.571478 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.573715 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.574452 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.574575 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.574735 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.575038 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.575138 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
I20260929 10:25:57.575773 3562 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: No bootstrap required, opened a new log
I20260929 10:25:57.575865 3562 ts_tablet_manager.cc:1397] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 10:25:57.575901 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.568915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:43981 (local address 127.0.0.1:40204)
0929 10:25:57.569086 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.569100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.569176 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.570251 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.570261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.570277 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:57.570380 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.570387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.571776 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.571779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.573244 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.573255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.575500 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.575503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.575622 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.575649 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.575697 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.575769 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":76}
I20260929 10:25:57.576282 3562 raft_consensus.cc:348] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.576401 3562 raft_consensus.cc:374] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:57.576438 3562 raft_consensus.cc:729] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fca613ef1bd41beb22e533afc421d45, State: Initialized, Role: FOLLOWER
I20260929 10:25:57.576488 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.569067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.21.33.131:43981)
0929 10:25:57.569779 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.569791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.569814 (+ 23us) server_negotiation.cc:407] Connection header received
0929 10:25:57.569884 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.569889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.570093 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.570368 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:57.570530 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.571624 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.573365 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.575363 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.576088 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.576224 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.576274 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.576338 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":206,"threads_started":1}
I20260929 10:25:57.576577 3562 consensus_queue.cc:260] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.576812 3562 ts_tablet_manager.cc:1428] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:25:57.578508 3742 tablet_service.cc:1467] Processing CreateTablet for tablet 715bdfd22c3e417aa1001efddb937a9e (DEFAULT_TABLE table=test-table [id=5ebe4e3604e84a73aaa40b81291e2cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:57.579356 3742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715bdfd22c3e417aa1001efddb937a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:57.579164 3874 tablet_service.cc:1467] Processing CreateTablet for tablet 715bdfd22c3e417aa1001efddb937a9e (DEFAULT_TABLE table=test-table [id=5ebe4e3604e84a73aaa40b81291e2cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:25:57.579993 3874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 715bdfd22c3e417aa1001efddb937a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:25:57.586462 3953 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Bootstrap starting.
I20260929 10:25:57.586493 3954 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Bootstrap starting.
I20260929 10:25:57.588455 3953 tablet_bootstrap.cc:654] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:57.588856 3954 tablet_bootstrap.cc:654] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Neither blocks nor log segments found. Creating new log.
I20260929 10:25:57.589568 3953 log.cc:824] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:57.589569 3954 log.cc:824] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:57.591310 3954 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: No bootstrap required, opened a new log
I20260929 10:25:57.591318 3953 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: No bootstrap required, opened a new log
I20260929 10:25:57.591504 3954 ts_tablet_manager.cc:1397] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:25:57.591519 3953 ts_tablet_manager.cc:1397] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:25:57.594250 3953 raft_consensus.cc:348] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.594250 3954 raft_consensus.cc:348] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.594455 3953 raft_consensus.cc:374] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:57.594485 3954 raft_consensus.cc:374] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:25:57.594487 3953 raft_consensus.cc:729] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad6c5f036094c74aca94261faf54512, State: Initialized, Role: FOLLOWER
I20260929 10:25:57.594550 3954 raft_consensus.cc:729] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c74fd95846e4dec9a27d2c8db81d787, State: Initialized, Role: FOLLOWER
I20260929 10:25:57.594942 3953 consensus_queue.cc:260] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.594941 3954 consensus_queue.cc:260] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.595566 3808 heartbeater.cc:503] Master 127.21.33.190:35163 was elected leader, sending a full tablet report...
I20260929 10:25:57.595937 3953 ts_tablet_manager.cc:1428] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 10:25:57.596272 3954 ts_tablet_manager.cc:1428] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:25:57.596532 3940 heartbeater.cc:503] Master 127.21.33.190:35163 was elected leader, sending a full tablet report...
W20260929 10:25:57.615068 3671 tablet.cc:2369] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:25:57.662303 3958 raft_consensus.cc:482] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:25:57.662528 3958 raft_consensus.cc:504] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.663686 3958 leader_election.cc:288] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981), 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797)
I20260929 10:25:57.668035 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.663965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:43981 (local address 127.21.33.130:56933)
0929 10:25:57.664092 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.664108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.664205 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.664481 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.664483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.664506 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:57.664633 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.664639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.665644 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.665646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.666868 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.666878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.667683 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.667686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.667766 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.667785 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.667846 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.667918 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":27}
I20260929 10:25:57.668133 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.664060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:56933 (local address 127.21.33.131:43981)
0929 10:25:57.664181 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.664185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.664216 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:25:57.664303 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.664306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.664354 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.664450 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:57.664756 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.665523 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.667015 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.667581 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.667696 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.667747 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.667877 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.667945 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:25:57.668536 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.664248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44797 (local address 127.21.33.130:45193)
0929 10:25:57.664673 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.664687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:57.664767 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:57.665061 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:57.665064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:57.665082 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:25:57.665183 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.665189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.666062 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.666065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.667186 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:57.667194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.668132 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.668136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:57.668245 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.668277 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 10:25:57.668341 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 10:25:57.668401 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":18,"thread_start_us":149,"threads_started":1}
I20260929 10:25:57.668748 3894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715bdfd22c3e417aa1001efddb937a9e" candidate_uuid: "2c74fd95846e4dec9a27d2c8db81d787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad6c5f036094c74aca94261faf54512" is_pre_election: true
I20260929 10:25:57.668897 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.664364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:45193 (local address 127.21.33.129:44797)
0929 10:25:57.664636 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.664640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.664755 (+ 115us) server_negotiation.cc:407] Connection header received
0929 10:25:57.664845 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.664848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.664911 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.665029 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:25:57.665355 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.665948 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.667328 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.667985 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.668106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.668270 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.668690 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.668773 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260929 10:25:57.669098 3894 raft_consensus.cc:2454] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c74fd95846e4dec9a27d2c8db81d787 in term 0.
I20260929 10:25:57.669420 3624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715bdfd22c3e417aa1001efddb937a9e" candidate_uuid: "2c74fd95846e4dec9a27d2c8db81d787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fca613ef1bd41beb22e533afc421d45" is_pre_election: true
I20260929 10:25:57.669618 3698 leader_election.cc:302] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c74fd95846e4dec9a27d2c8db81d787, 8ad6c5f036094c74aca94261faf54512; no voters:
I20260929 10:25:57.669708 3624 raft_consensus.cc:2454] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c74fd95846e4dec9a27d2c8db81d787 in term 0.
I20260929 10:25:57.669893 3958 raft_consensus.cc:2781] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:25:57.669987 3958 raft_consensus.cc:482] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:25:57.670047 3958 raft_consensus.cc:3037] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:57.671072 3958 raft_consensus.cc:504] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.671404 3958 leader_election.cc:288] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [CANDIDATE]: Term 1 election: Requested vote from peers 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981), 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797)
I20260929 10:25:57.671813 3894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715bdfd22c3e417aa1001efddb937a9e" candidate_uuid: "2c74fd95846e4dec9a27d2c8db81d787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ad6c5f036094c74aca94261faf54512"
I20260929 10:25:57.671813 3624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715bdfd22c3e417aa1001efddb937a9e" candidate_uuid: "2c74fd95846e4dec9a27d2c8db81d787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fca613ef1bd41beb22e533afc421d45"
I20260929 10:25:57.671942 3894 raft_consensus.cc:3037] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:57.671942 3624 raft_consensus.cc:3037] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:25:57.673182 3894 raft_consensus.cc:2454] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c74fd95846e4dec9a27d2c8db81d787 in term 1.
I20260929 10:25:57.673182 3624 raft_consensus.cc:2454] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c74fd95846e4dec9a27d2c8db81d787 in term 1.
I20260929 10:25:57.673455 3698 leader_election.cc:302] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c74fd95846e4dec9a27d2c8db81d787, 4fca613ef1bd41beb22e533afc421d45; no voters:
I20260929 10:25:57.673633 3958 raft_consensus.cc:2781] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:25:57.673846 3958 raft_consensus.cc:686] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 LEADER]: Becoming Leader. State: Replica: 2c74fd95846e4dec9a27d2c8db81d787, State: Running, Role: LEADER
I20260929 10:25:57.674211 3958 consensus_queue.cc:237] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:57.677088 3482 catalog_manager.cc:5495] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130). New cstate: current_term: 1 leader_uuid: "2c74fd95846e4dec9a27d2c8db81d787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } health_report { overall_health: UNKNOWN } } }
I20260929 10:25:57.701505 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.698160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.21.33.130:33799)
0929 10:25:57.698291 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:57.698294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:57.698307 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:25:57.698407 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:57.698410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:57.698464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:57.698553 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:25:57.698762 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.699267 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.700041 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:57.700255 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:57.700343 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:57.700587 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:25:57.701043 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:25:57.701136 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:57.701259 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 10:25:57.701321 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:25:57.707297 3894 raft_consensus.cc:1261] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 1 FOLLOWER]: Refusing update from remote peer 2c74fd95846e4dec9a27d2c8db81d787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:57.707295 3624 raft_consensus.cc:1261] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Refusing update from remote peer 2c74fd95846e4dec9a27d2c8db81d787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:25:57.707994 3958 consensus_queue.cc:1035] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:57.708168 3961 consensus_queue.cc:1035] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:25:57.714717 3965 mvcc.cc:205] Tried to move back new op lower bound from 7334615276361707520 to 7334615276235890688. Current Snapshot: MvccSnapshot[applied={T|T < 7334615276361707520}]
I20260929 10:25:57.723760 3481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:57.723919 3481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:25:57.726387 3481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Sending DeleteTablet for 3 replicas of tablet 715bdfd22c3e417aa1001efddb937a9e
I20260929 10:25:57.727200 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 3520
I20260929 10:25:57.727152 3874 tablet_service.cc:1514] Processing DeleteTablet for tablet 715bdfd22c3e417aa1001efddb937a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:57 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:40204
I20260929 10:25:57.727383 3742 tablet_service.cc:1514] Processing DeleteTablet for tablet 715bdfd22c3e417aa1001efddb937a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:25:57 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:37156
I20260929 10:25:57.728960 3974 tablet_replica.cc:330] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: stopping tablet replica
I20260929 10:25:57.729259 3974 raft_consensus.cc:2229] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:25:57.729743 3974 raft_consensus.cc:2258] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:25:57.732900 3974 ts_tablet_manager.cc:1905] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:57.733412 3973 tablet_replica.cc:330] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: stopping tablet replica
I20260929 10:25:57.733789 3973 raft_consensus.cc:2229] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:25:57.734171 3973 raft_consensus.cc:2258] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 10:25:57.735260 3469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:44797 (error 108)
I20260929 10:25:57.735778 3973 ts_tablet_manager.cc:1905] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:57.736428 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 3674
I20260929 10:25:57.736522 3974 ts_tablet_manager.cc:1918] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:57.736629 3974 log.cc:1196] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/715bdfd22c3e417aa1001efddb937a9e
I20260929 10:25:57.738590 3973 ts_tablet_manager.cc:1918] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:57.738745 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.738356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44797 (local address 127.0.0.1:45190)
0929 10:25:57.738553 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.738641 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:44797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 10:25:57.738781 3973 log.cc:1196] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/715bdfd22c3e417aa1001efddb937a9e
I20260929 10:25:57.739132 3973 ts_tablet_manager.cc:1939] T 715bdfd22c3e417aa1001efddb937a9e P 8ad6c5f036094c74aca94261faf54512: Deleting consensus metadata
W20260929 10:25:57.738950 3469 catalog_manager.cc:4584] TS 4fca613ef1bd41beb22e533afc421d45 (127.21.33.129:44797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 715bdfd22c3e417aa1001efddb937a9e: Network error: Client connection negotiation failed: client connection to 127.21.33.129:44797: connect: Connection refused (error 111)
I20260929 10:25:57.740288 3469 catalog_manager.cc:4841] TS 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981): tablet 715bdfd22c3e417aa1001efddb937a9e (table test-table [id=5ebe4e3604e84a73aaa40b81291e2cf7]) successfully deleted
I20260929 10:25:57.743595 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 3811
I20260929 10:25:57.744026 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:57.743712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33799 (local address 127.0.0.1:37174)
0929 10:25:57.743884 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:57.743937 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.130:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 10:25:57.749462 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 3449
I20260929 10:25:57.756255 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35163
--webserver_interface=127.21.33.190
--webserver_port=40025
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:35163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:25:57.866097 3976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:57.866415 3976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:57.866496 3976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:57.870543 3976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:25:57.870628 3976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:57.870668 3976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:57.870721 3976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:25:57.870761 3976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:25:57.876080 3976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:35163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=40025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:57.877140 3976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:57.878255 3976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:25:57.886998 3976 server_base.cc:1034] running on GCE node
W20260929 10:25:57.886942 3981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:57.886963 3984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:57.886942 3982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:57.887773 3976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:57.888927 3976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:57.890111 3976 hybrid_clock.cc:648] HybridClock initialized: now 1790677557890064 us; error 60 us; skew 500 ppm
Sep 29 10:25:57 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:57.892944 3976 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:25:57.893834 3976 webserver.cc:460] Webserver started at http://127.21.33.190:40025/ using document root <none> and password file <none>
I20260929 10:25:57.894357 3976 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:57.894418 3976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:57.898177 3976 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:25:57.900218 3991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:57.901681 3976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:25:57.901800 3976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "6b7f9269fa8446e1b7376d9da4d90238"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "1097026da817641a5a37980558827993"
server_key_iv: "3c77b2bcb145a43cba59ddd46ea19b04"
server_key_version: "encryptionkey@0"
I20260929 10:25:57.902426 3976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:57.921284 3976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:57.924901 3976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:57.925083 3976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:57.932638 3976 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:35163
I20260929 10:25:57.932945 4043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:35163 every 8 connection(s)
I20260929 10:25:57.933866 3976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:25:57.938336 4044 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:25:57.940213 4044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Bootstrap starting.
I20260929 10:25:57.942837 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 3976
I20260929 10:25:57.943398 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:44797
--local_ip_for_outbound_sockets=127.21.33.129
--tserver_master_addrs=127.21.33.190:35163
--webserver_port=34951
--webserver_interface=127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:57.958600 4044 log.cc:824] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:57.970633 4044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:25:57.971042 4044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238: Bootstrap complete.
I20260929 10:25:57.975128 4044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:57.975518 4044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7f9269fa8446e1b7376d9da4d90238, State: Initialized, Role: FOLLOWER
I20260929 10:25:57.976004 4044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:57.976115 4044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:25:57.976195 4044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:25:57.976310 4044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:25:57.978420 4044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:57.978811 4044 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7f9269fa8446e1b7376d9da4d90238; no voters:
I20260929 10:25:57.979171 4044 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 10:25:57.979255 4048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:25:57.979435 4048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [term 3 LEADER]: Becoming Leader. State: Replica: 6b7f9269fa8446e1b7376d9da4d90238, State: Running, Role: LEADER
I20260929 10:25:57.979723 4048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } }
I20260929 10:25:57.980420 4044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:25:57.980672 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7f9269fa8446e1b7376d9da4d90238. Latest consensus state: current_term: 3 leader_uuid: "6b7f9269fa8446e1b7376d9da4d90238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } } }
I20260929 10:25:57.980779 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:57.981238 4049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6b7f9269fa8446e1b7376d9da4d90238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7f9269fa8446e1b7376d9da4d90238" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 35163 } } }
I20260929 10:25:57.981427 4049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7f9269fa8446e1b7376d9da4d90238 [sys.catalog]: This master's current role is: LEADER
I20260929 10:25:57.982903 4052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:25:57.988534 4052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:25:57.989298 4052 catalog_manager.cc:1259] Loaded cluster ID: 64b4e814742d4a24b25677814443ba56
I20260929 10:25:57.989363 4052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 10:25:57.990480 4052 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffae27109ad google::LogMessage::Fail() at ??:0
@ 0x7ffae271388a google::LogMessage::SendToLog() at ??:0
@ 0x7ffae27104de google::LogMessage::Flush() at ??:0
@ 0x7ffae2711a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffaed04ffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ffaed050d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ffaed04d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ffaed091640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ffae49623d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffae4962c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ffae49643f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ffae4953512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffae5f976db start_thread at ??:0
@ 0x7ffae120e61f clone at ??:0
W20260929 10:25:58.076854 4046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:58.077149 4046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:58.077262 4046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:58.081561 4046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:58.081645 4046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:58.081756 4046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:25:58.087019 4046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:44797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=34951
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:58.088095 4046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:58.089124 4046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:58.099954 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:58.099946 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:58.099946 4088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:58.100415 4046 server_base.cc:1034] running on GCE node
I20260929 10:25:58.100831 4046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:58.101498 4046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:58.102684 4046 hybrid_clock.cc:648] HybridClock initialized: now 1790677558102654 us; error 48 us; skew 500 ppm
Sep 29 10:25:58 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677558, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:58.105324 4046 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:25:58.106125 4046 webserver.cc:460] Webserver started at http://127.21.33.129:34951/ using document root <none> and password file <none>
I20260929 10:25:58.106673 4046 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:58.106760 4046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:58.110659 4046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:25:58.112839 4095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:58.114140 4046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:25:58.114262 4046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "4fca613ef1bd41beb22e533afc421d45"
format_stamp: "Formatted at 2026-09-29 10:25:55 on dist-test-slave-2rbl"
server_key: "61d1653ddad7446a926f4fc7bfa2466d"
server_key_iv: "9b7cb7e4093be191e7aaa50e76df9596"
server_key_version: "encryptionkey@0"
I20260929 10:25:58.114791 4046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:58.131247 4046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:58.135514 4046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:58.135727 4046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:58.136457 4046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:58.137823 4102 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:25:58.142005 4046 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:25:58.142082 4046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260929 10:25:58.142149 4046 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:25:58.144290 4046 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:25:58.144348 4046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 10:25:58.144731 4102 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Bootstrap starting.
I20260929 10:25:58.155385 4046 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:44797
I20260929 10:25:58.156610 4046 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:25:58.157637 4209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:44797 every 8 connection(s)
I20260929 10:25:58.160425 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 4046
I20260929 10:25:58.160897 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:33799
--local_ip_for_outbound_sockets=127.21.33.130
--tserver_master_addrs=127.21.33.190:35163
--webserver_port=39917
--webserver_interface=127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:58.164899 4102 log.cc:824] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Log is configured to *not* fsync() on all Append() calls
I20260929 10:25:58.171555 4102 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:25:58.171953 4102 tablet_bootstrap.cc:492] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Bootstrap complete.
I20260929 10:25:58.173062 4102 ts_tablet_manager.cc:1397] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20260929 10:25:58.175407 4102 raft_consensus.cc:348] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:58.175902 4102 raft_consensus.cc:729] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fca613ef1bd41beb22e533afc421d45, State: Initialized, Role: FOLLOWER
I20260929 10:25:58.176394 4102 consensus_queue.cc:260] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:58.177420 4102 ts_tablet_manager.cc:1428] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260929 10:25:58.231262 4212 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:25:58.158987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:55785)
0929 10:25:58.159604 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.159685 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:58.160861 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:58.230458 (+ 69597us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":154,"threads_started":1}
W20260929 10:25:58.231889 4210 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:35163 (0 consecutive failures): Network error: Failed to ping master at 127.21.33.190:35163: Client connection negotiation failed: client connection to 127.21.33.190:35163: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:25:58.232643 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.232382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:41357)
0929 10:25:58.232507 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.232584 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 10:25:58.233337 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.233117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:34607)
0929 10:25:58.233241 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.233284 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 10:25:58.233603 4210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:25:58.296341 4213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:58.296597 4213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:58.296640 4213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:58.300849 4213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:58.300907 4213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:58.301002 4213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:25:58.306226 4213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:33799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=39917
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:58.307233 4213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:58.308211 4213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:58.317582 4220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:58.317600 4223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:58.317530 4221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:58.317835 4213 server_base.cc:1034] running on GCE node
I20260929 10:25:58.318270 4213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:58.318912 4213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:58.320113 4213 hybrid_clock.cc:648] HybridClock initialized: now 1790677558320078 us; error 45 us; skew 500 ppm
Sep 29 10:25:58 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677558, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:58.322969 4213 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:25:58.323926 4213 webserver.cc:460] Webserver started at http://127.21.33.130:39917/ using document root <none> and password file <none>
I20260929 10:25:58.324551 4213 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:58.324648 4213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:58.328780 4213 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:25:58.330909 4230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:58.332245 4213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:25:58.332401 4213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "2c74fd95846e4dec9a27d2c8db81d787"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "3d652ccd1f93f6e3bf2f7f17b83074aa"
server_key_iv: "1029818b090e1836f83a3c576099d562"
server_key_version: "encryptionkey@0"
I20260929 10:25:58.333010 4213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:58.361395 4213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:58.364737 4213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:58.364943 4213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:58.365574 4213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:58.366640 4237 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 10:25:58.370581 4237 ts_tablet_manager.cc:1852] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 10:25:58.370977 4237 ts_tablet_manager.cc:1905] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:25:58.374866 4237 ts_tablet_manager.cc:1918] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:25:58.375044 4237 ts_tablet_manager.cc:1939] T 715bdfd22c3e417aa1001efddb937a9e P 2c74fd95846e4dec9a27d2c8db81d787: Deleting consensus metadata
I20260929 10:25:58.375553 4213 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 10:25:58.375646 4213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260929 10:25:58.375730 4213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:58.375769 4213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:58.384760 4213 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:33799
I20260929 10:25:58.384826 4344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:33799 every 8 connection(s)
I20260929 10:25:58.385712 4213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:25:58.389107 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.387816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:34261)
0929 10:25:58.388286 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.388396 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
W20260929 10:25:58.389595 4345 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:35163 (0 consecutive failures): Network error: Failed to ping master at 127.21.33.190:35163: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
I20260929 10:25:58.390187 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.389932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:46445)
0929 10:25:58.390055 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.390119 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 10:25:58.390192 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 4213
I20260929 10:25:58.390617 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:43981
--local_ip_for_outbound_sockets=127.21.33.131
--tserver_master_addrs=127.21.33.190:35163
--webserver_port=44845
--webserver_interface=127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:25:58.390864 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.390619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:35285)
0929 10:25:58.390733 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.390794 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:25:58.391070 4345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:25:58.499871 4348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:25:58.500129 4348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:25:58.500213 4348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:25:58.504215 4348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:25:58.504273 4348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:25:58.504369 4348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:25:58.509516 4348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:43981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=44845
--tserver_master_addrs=127.21.33.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:25:58.510564 4348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:25:58.511586 4348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:25:58.521183 4354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:58.521296 4348 server_base.cc:1034] running on GCE node
W20260929 10:25:58.521179 4356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:25:58.521222 4353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:25:58.521965 4348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:25:58.522635 4348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:25:58.523826 4348 hybrid_clock.cc:648] HybridClock initialized: now 1790677558523791 us; error 47 us; skew 500 ppm
Sep 29 10:25:58 dist-test-slave-2rbl krb5kdc[2902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677558, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:25:58.526742 4348 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:25:58.527709 4348 webserver.cc:460] Webserver started at http://127.21.33.131:44845/ using document root <none> and password file <none>
I20260929 10:25:58.528314 4348 fs_manager.cc:359] Metadata directory not provided
I20260929 10:25:58.528404 4348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:25:58.532332 4348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:25:58.534503 4363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:58.535883 4348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:25:58.536029 4348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "8ad6c5f036094c74aca94261faf54512"
format_stamp: "Formatted at 2026-09-29 10:25:56 on dist-test-slave-2rbl"
server_key: "7f16c990939d17b69ec92d59766275f8"
server_key_iv: "f7defed540557d47437b6774547c989e"
server_key_version: "encryptionkey@0"
I20260929 10:25:58.536593 4348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:25:58.548465 4348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:25:58.551694 4348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:25:58.551883 4348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:25:58.552438 4348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:25:58.553274 4348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:25:58.553334 4348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:58.553404 4348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:25:58.553449 4348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:25:58.562282 4348 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:43981
I20260929 10:25:58.562306 4476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:43981 every 8 connection(s)
I20260929 10:25:58.563328 4348 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:25:58.566893 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 4348
I20260929 10:25:58.567338 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.565494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:53147)
0929 10:25:58.565959 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.566377 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
W20260929 10:25:58.567858 4477 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:35163 (0 consecutive failures): Network error: Failed to ping master at 127.21.33.190:35163: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
I20260929 10:25:58.568452 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.568256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:32781)
0929 10:25:58.568355 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.568398 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 10:25:58.568945 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:58.568782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:42723)
0929 10:25:58.568865 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:58.568902 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 10:25:58.569154 4477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:25:59.234647 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:59.234202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:44505)
0929 10:25:59.234444 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:59.234506 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260929 10:25:59.392031 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:59.391639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:44935)
0929 10:25:59.391877 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:59.391934 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260929 10:25:59.570211 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:59.569751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:47297)
0929 10:25:59.570047 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:59.570107 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260929 10:25:59.661762 4484 raft_consensus.cc:482] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:25:59.661983 4484 raft_consensus.cc:504] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:25:59.663141 4484 leader_election.cc:288] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981), 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
Sep 29 10:25:59 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677558, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.130@KRBTEST.COM
Sep 29 10:25:59 dist-test-slave-2rbl krb5kdc[2902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677558, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.131@KRBTEST.COM
I20260929 10:25:59.680926 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:59.663348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:43981 (local address 127.21.33.129:45199)
0929 10:25:59.663534 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:59.663547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:59.663653 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:59.665904 (+ 2251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:59.665911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:59.666584 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:59.667452 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:59.667463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.669483 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.669487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:59.671175 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:59.671188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.672173 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.672178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:59.672514 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:59.673347 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:59.673376 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:59.675980 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:59.678438 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:59.678442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:59.678447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:59.678679 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:59.679026 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:59.679029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:59.679030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:59.679118 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:59.680120 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:59.680123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:59.680217 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 10:25:59.680354 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 10:25:59.680721 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":411,"spinlock_wait_cycles":36608}
I20260929 10:25:59.680927 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:59.663550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45199 (local address 127.21.33.131:43981)
0929 10:25:59.663914 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:59.663944 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:59.664022 (+ 78us) server_negotiation.cc:407] Connection header received
0929 10:25:59.664925 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:59.664934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:59.665443 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:59.665781 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:59.667666 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.669324 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.671314 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.672047 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.672192 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:59.676184 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:59.676188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:59.676198 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:59.676253 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:59.678340 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:59.678786 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:59.678791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:59.678795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:59.678930 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:59.679244 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:59.679247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:59.679248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:59.679997 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:59.680125 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:59.680467 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 10:25:59.680730 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260929 10:25:59.680955 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:59.663708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:35417 (local address 127.21.33.130:33799)
0929 10:25:59.664053 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 10:25:59.664067 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:25:59.664098 (+ 31us) server_negotiation.cc:407] Connection header received
0929 10:25:59.664977 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:25:59.664985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:25:59.665442 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:25:59.665780 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:25:59.667666 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.669324 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.671314 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.672019 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.672192 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:59.675707 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:25:59.675712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:25:59.675724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:25:59.675750 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:25:59.678047 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:59.678612 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:59.678617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:59.678621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:59.678782 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:25:59.679194 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:25:59.679198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:25:59.679199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:25:59.679757 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:25:59.679883 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 10:25:59.680487 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 10:25:59.680759 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":131,"threads_started":1}
I20260929 10:25:59.680991 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:25:59.663396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33799 (local address 127.21.33.129:35417)
0929 10:25:59.663677 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 10:25:59.663691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:25:59.663786 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:25:59.665904 (+ 2118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:25:59.665911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:25:59.666584 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:25:59.667440 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:59.667463 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.669485 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.669488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:59.671175 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:25:59.671188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:25:59.672173 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:25:59.672178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:25:59.672514 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:25:59.673347 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:25:59.673392 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:25:59.675472 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:25:59.678170 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:59.678175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:59.678186 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:59.678500 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:59.678880 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:25:59.678882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:25:59.678884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:25:59.679063 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:25:59.679952 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:25:59.679958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:25:59.680173 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 10:25:59.680355 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 10:25:59.680805 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":538,"spinlock_wait_cycles":32768,"thread_start_us":86,"threads_started":1}
W20260929 10:25:59.683542 4099 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
W20260929 10:25:59.683755 4098 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
I20260929 10:25:59.683863 4098 leader_election.cc:302] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fca613ef1bd41beb22e533afc421d45; no voters: 2c74fd95846e4dec9a27d2c8db81d787, 8ad6c5f036094c74aca94261faf54512
I20260929 10:25:59.684173 4484 raft_consensus.cc:2726] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:26:00.236024 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:00.235462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:52839)
0929 10:26:00.235767 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:00.235832 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
I20260929 10:26:00.393136 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:00.392746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:39471)
0929 10:26:00.392980 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:00.393037 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260929 10:26:00.571384 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:00.570946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:34149)
0929 10:26:00.571230 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:00.571294 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260929 10:26:01.237548 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:01.236982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:37177)
0929 10:26:01.237380 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:01.237444 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":78,"threads_started":1}
I20260929 10:26:01.268281 4492 raft_consensus.cc:482] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:26:01.268412 4492 raft_consensus.cc:504] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:26:01.268839 4492 leader_election.cc:288] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981), 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
W20260929 10:26:01.269685 4098 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
W20260929 10:26:01.269775 4099 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
I20260929 10:26:01.269896 4099 leader_election.cc:302] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fca613ef1bd41beb22e533afc421d45; no voters: 2c74fd95846e4dec9a27d2c8db81d787, 8ad6c5f036094c74aca94261faf54512
I20260929 10:26:01.270088 4492 raft_consensus.cc:2726] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:26:01.394472 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:01.393974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:42413)
0929 10:26:01.394271 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:01.394339 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260929 10:26:01.572772 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:01.572290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:39831)
0929 10:26:01.572590 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:01.572653 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260929 10:26:02.238842 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:02.238384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:45603)
0929 10:26:02.238656 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:02.238742 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
I20260929 10:26:02.396019 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:02.395549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:58023)
0929 10:26:02.395831 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:02.395894 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260929 10:26:02.574218 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:02.573760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:42373)
0929 10:26:02.574036 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:02.574123 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260929 10:26:03.240324 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:03.239845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:49315)
0929 10:26:03.240171 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:03.240230 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260929 10:26:03.397492 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:03.396946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:39709)
0929 10:26:03.397331 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:03.397394 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260929 10:26:03.575577 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:03.575134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:55331)
0929 10:26:03.575413 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:03.575477 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260929 10:26:04.034590 4501 raft_consensus.cc:482] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:26:04.034724 4501 raft_consensus.cc:504] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:26:04.035156 4501 leader_election.cc:288] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981), 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
W20260929 10:26:04.035961 4098 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
W20260929 10:26:04.036104 4099 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
I20260929 10:26:04.036211 4099 leader_election.cc:302] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fca613ef1bd41beb22e533afc421d45; no voters: 2c74fd95846e4dec9a27d2c8db81d787, 8ad6c5f036094c74aca94261faf54512
I20260929 10:26:04.036377 4501 raft_consensus.cc:2726] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:26:04.241485 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:04.241036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:38861)
0929 10:26:04.241338 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:04.241393 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":86,"threads_started":1}
I20260929 10:26:04.398798 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:04.398319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:42213)
0929 10:26:04.398611 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:04.398673 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260929 10:26:04.576905 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:04.576466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:40391)
0929 10:26:04.576755 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:04.576816 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260929 10:26:05.242743 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:05.242298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:45615)
0929 10:26:05.242567 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:05.242633 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260929 10:26:05.400156 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:05.399701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:42123)
0929 10:26:05.400007 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:05.400061 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":77,"threads_started":1}
I20260929 10:26:05.578183 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:05.577691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:54047)
0929 10:26:05.578009 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:05.578067 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":71,"threads_started":1}
I20260929 10:26:06.244120 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:06.243629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:56585)
0929 10:26:06.243938 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:06.244015 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":77,"threads_started":1}
I20260929 10:26:06.393471 4509 raft_consensus.cc:482] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:26:06.393610 4509 raft_consensus.cc:504] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:26:06.394114 4509 leader_election.cc:288] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981), 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
W20260929 10:26:06.394840 4098 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
W20260929 10:26:06.394966 4099 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
I20260929 10:26:06.395073 4099 leader_election.cc:302] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fca613ef1bd41beb22e533afc421d45; no voters: 2c74fd95846e4dec9a27d2c8db81d787, 8ad6c5f036094c74aca94261faf54512
I20260929 10:26:06.395255 4509 raft_consensus.cc:2726] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:26:06.401338 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:06.400837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:36585)
0929 10:26:06.401111 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:06.401165 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":69,"threads_started":1}
I20260929 10:26:06.579422 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:06.579018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:41875)
0929 10:26:06.579270 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:06.579329 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260929 10:26:07.245474 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:07.244983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:49839)
0929 10:26:07.245318 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:07.245380 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":81,"threads_started":1}
I20260929 10:26:07.402611 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:07.402120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:56023)
0929 10:26:07.402389 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:07.402447 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260929 10:26:07.580775 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:07.580299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:59849)
0929 10:26:07.580613 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:07.580679 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260929 10:26:08.246850 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:08.246359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:59713)
0929 10:26:08.246664 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:08.246733 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260929 10:26:08.403837 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:08.403396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:53591)
0929 10:26:08.403681 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:08.403745 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260929 10:26:08.582227 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:08.581753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:55115)
0929 10:26:08.582062 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:08.582130 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":72,"threads_started":1}
I20260929 10:26:09.248129 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:09.247687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:58713)
0929 10:26:09.247976 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:09.248034 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260929 10:26:09.405182 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:09.404736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:55227)
0929 10:26:09.405022 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:09.405085 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260929 10:26:09.583532 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:09.583087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:57691)
0929 10:26:09.583358 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:09.583418 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260929 10:26:10.249557 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:10.249010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:58937)
0929 10:26:10.249381 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:10.249459 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":89,"threads_started":1}
I20260929 10:26:10.406523 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:10.406018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:42349)
0929 10:26:10.406292 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:10.406358 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260929 10:26:10.584810 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:10.584358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:51531)
0929 10:26:10.584650 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:10.584705 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260929 10:26:11.250730 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:11.250304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:43549)
0929 10:26:11.250578 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:11.250633 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":72,"threads_started":1}
I20260929 10:26:11.407809 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:11.407390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:54937)
0929 10:26:11.407659 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:11.407717 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260929 10:26:11.586087 4526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:11.585674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:41101)
0929 10:26:11.585930 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:11.585987 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260929 10:26:12.251904 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:12.251497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:51937)
0929 10:26:12.251749 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:12.251808 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
I20260929 10:26:12.409279 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:12.408745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:33851)
0929 10:26:12.409070 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:12.409139 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260929 10:26:12.587450 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:12.586962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.131:52347)
0929 10:26:12.587290 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:12.587354 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260929 10:26:12.929922 4530 raft_consensus.cc:482] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:26:12.930106 4530 raft_consensus.cc:504] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ad6c5f036094c74aca94261faf54512" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 43981 } } peers { permanent_uuid: "2c74fd95846e4dec9a27d2c8db81d787" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33799 } } peers { permanent_uuid: "4fca613ef1bd41beb22e533afc421d45" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44797 } }
I20260929 10:26:12.930675 4530 leader_election.cc:288] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981), 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799)
W20260929 10:26:12.931501 4099 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8ad6c5f036094c74aca94261faf54512 (127.21.33.131:43981): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
W20260929 10:26:12.931596 4098 leader_election.cc:341] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c74fd95846e4dec9a27d2c8db81d787 (127.21.33.130:33799): Not found: Tablet not found: 715bdfd22c3e417aa1001efddb937a9e
I20260929 10:26:12.931643 4099 leader_election.cc:302] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fca613ef1bd41beb22e533afc421d45; no voters: 2c74fd95846e4dec9a27d2c8db81d787, 8ad6c5f036094c74aca94261faf54512
I20260929 10:26:12.931802 4530 raft_consensus.cc:2726] T 715bdfd22c3e417aa1001efddb937a9e P 4fca613ef1bd41beb22e533afc421d45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:26:13.253367 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:13.252885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.129:57535)
0929 10:26:13.253156 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:13.253267 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260929 10:26:13.410703 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:13.410286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:35163 (local address 127.21.33.130:47337)
0929 10:26:13.410543 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:13.410608 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260929 10:26:13.568759 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 4046
I20260929 10:26:13.574568 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 4213
I20260929 10:26:13.579928 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 4348
2026-09-29T10:26:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19238 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:26:13 dist-test-slave-2rbl krb5kdc[4535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:26:13 dist-test-slave-2rbl krb5kdc[4535](info): set up 2 sockets
Sep 29 10:26:13 dist-test-slave-2rbl krb5kdc[4535](info): commencing operation
krb5kdc: starting...
W20260929 10:26:14.622371 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:26:14 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:26:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:26:14.752099 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46279
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:46279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:26:14.862761 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:14.863071 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:14.863157 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:14.867200 4548 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:26:14.867296 4548 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:26:14.867324 4548 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:26:14.867357 4548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:26:14.867396 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:14.867434 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:14.867491 4548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:26:14.867525 4548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:26:14.872794 4548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:46279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:14.873957 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:14.875038 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:14.884195 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:14.884156 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:14.884195 4548 server_base.cc:1034] running on GCE node
W20260929 10:26:14.884177 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:14.885305 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:14.886466 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:14.887653 4548 hybrid_clock.cc:648] HybridClock initialized: now 1790677574887615 us; error 52 us; skew 500 ppm
Sep 29 10:26:14 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677574, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:14.890811 4548 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:26:14.891763 4548 webserver.cc:460] Webserver started at http://127.21.33.190:46719/ using document root <none> and password file <none>
I20260929 10:26:14.892355 4548 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:14.892422 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:14.892685 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:26:14.894856 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "3e98fefa941c4c2a9aadb17d3c04b396"
format_stamp: "Formatted at 2026-09-29 10:26:14 on dist-test-slave-2rbl"
server_key: "ef85f814435eec64856cc1d2eef67227"
server_key_iv: "5ae54d52dfa2f0e14552f825e5cc4685"
server_key_version: "encryptionkey@0"
I20260929 10:26:14.895439 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "3e98fefa941c4c2a9aadb17d3c04b396"
format_stamp: "Formatted at 2026-09-29 10:26:14 on dist-test-slave-2rbl"
server_key: "ef85f814435eec64856cc1d2eef67227"
server_key_iv: "5ae54d52dfa2f0e14552f825e5cc4685"
server_key_version: "encryptionkey@0"
I20260929 10:26:14.899447 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:26:14.902034 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:14.903481 4548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:26:14.903656 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "3e98fefa941c4c2a9aadb17d3c04b396"
format_stamp: "Formatted at 2026-09-29 10:26:14 on dist-test-slave-2rbl"
server_key: "ef85f814435eec64856cc1d2eef67227"
server_key_iv: "5ae54d52dfa2f0e14552f825e5cc4685"
server_key_version: "encryptionkey@0"
I20260929 10:26:14.903779 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:14.935117 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:14.938724 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:14.938963 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:14.946353 4548 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:46279
I20260929 10:26:14.946666 4615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:46279 every 8 connection(s)
I20260929 10:26:14.947360 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:26:14.949090 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 4548
I20260929 10:26:14.949265 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:26:14.949527 21638 external_mini_cluster.cc:1428] Setting key c5afd23e6974c64eaf46ebf8c4dc580d
I20260929 10:26:14.950379 4616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:26:14.956962 4616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Bootstrap starting.
Sep 29 10:26:14 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:26:14.959756 4616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Neither blocks nor log segments found. Creating new log.
I20260929 10:26:14.960525 4616 log.cc:824] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:14.962560 4616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: No bootstrap required, opened a new log
I20260929 10:26:14.964417 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:14.950865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.21.33.190:46279)
0929 10:26:14.951279 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:14.951295 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:14.951329 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:26:14.951965 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:14.951981 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:14.952303 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:14.952574 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:14.953571 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:14.954611 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:14.955834 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:14.956352 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:14.956488 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:14.959417 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:14.959423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:14.959437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:14.959466 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:14.961550 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:14.962013 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:14.962023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:14.962028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:14.962147 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:14.962448 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:14.962453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:14.962454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:14.962904 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:14.963102 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:14.963397 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 10:26:14.963692 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
I20260929 10:26:14.966650 4616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:14.966953 4616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:26:14.967026 4616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e98fefa941c4c2a9aadb17d3c04b396, State: Initialized, Role: FOLLOWER
I20260929 10:26:14.967460 4616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:14.967617 4616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:26:14.967685 4616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:26:14.967784 4616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:26:14.968726 4616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:14.969081 4616 leader_election.cc:302] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e98fefa941c4c2a9aadb17d3c04b396; no voters:
I20260929 10:26:14.969421 4616 leader_election.cc:288] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:26:14.969558 4621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:26:14.969813 4621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 LEADER]: Becoming Leader. State: Replica: 3e98fefa941c4c2a9aadb17d3c04b396, State: Running, Role: LEADER
I20260929 10:26:14.970175 4621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:14.970711 4616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:26:14.971700 4623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e98fefa941c4c2a9aadb17d3c04b396. Latest consensus state: current_term: 1 leader_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } } }
I20260929 10:26:14.971824 4623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: This master's current role is: LEADER
I20260929 10:26:14.972173 4622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } } }
I20260929 10:26:14.972324 4622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: This master's current role is: LEADER
I20260929 10:26:14.972379 4630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:26:14.975591 4630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:26:14.982010 4630 catalog_manager.cc:1347] Generated new cluster ID: 7b6290d5cf204bbf9878b4d23a061820
I20260929 10:26:14.982100 4630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:26:14.995236 4630 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:26:14.996493 4630 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:26:15.002900 4630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Generated new TSK 0
I20260929 10:26:15.003654 4630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:26:15.037010 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46279
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:26:15.150471 4642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:15.150735 4642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:15.150779 4642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:15.154944 4642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:15.155004 4642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:15.155093 4642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:26:15.160322 4642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:15.161448 4642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:15.162537 4642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:15.175184 4647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:15.175437 4648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:15.175184 4650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:15.175911 4642 server_base.cc:1034] running on GCE node
I20260929 10:26:15.176375 4642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:15.177055 4642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:15.178258 4642 hybrid_clock.cc:648] HybridClock initialized: now 1790677575178224 us; error 56 us; skew 500 ppm
Sep 29 10:26:15 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677575, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:15.181133 4642 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:26:15.182132 4642 webserver.cc:460] Webserver started at http://127.21.33.129:34021/ using document root <none> and password file <none>
I20260929 10:26:15.182670 4642 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:15.182734 4642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:15.182965 4642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:26:15.184999 4642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "32a87e189d25d7785d78ecb966e9e919"
server_key_iv: "b9f6299bca94afb9e227ace3c50b51d1"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.185626 4642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "32a87e189d25d7785d78ecb966e9e919"
server_key_iv: "b9f6299bca94afb9e227ace3c50b51d1"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.189479 4642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:26:15.192010 4657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.193447 4642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:26:15.193581 4642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "32a87e189d25d7785d78ecb966e9e919"
server_key_iv: "b9f6299bca94afb9e227ace3c50b51d1"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.193697 4642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:15.207645 4642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:15.211318 4642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:15.211537 4642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:15.212157 4642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:15.212985 4642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:26:15.213032 4642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.213110 4642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:26:15.213142 4642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.222520 4642 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:43079
I20260929 10:26:15.222537 4770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:43079 every 8 connection(s)
I20260929 10:26:15.223500 4642 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
Sep 29 10:26:15 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677575, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:26:15.233744 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 4642
I20260929 10:26:15.233897 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:26:15.234202 21638 external_mini_cluster.cc:1428] Setting key 18825432b70ffd527752c6934cc3c333
I20260929 10:26:15.238492 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.225533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:58751 (local address 127.21.33.190:46279)
0929 10:26:15.225706 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.225711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.226642 (+ 931us) server_negotiation.cc:407] Connection header received
0929 10:26:15.228132 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.228135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.228186 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.228268 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:15.229667 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.230239 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.231452 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.231712 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.231806 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.234506 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:15.234509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:15.234511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:15.234538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:15.236236 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.236792 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.236796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.236798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.236884 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.237303 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.237306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.237308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.237506 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:15.237629 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.238176 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.238316 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:26:15.239331 4773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.225947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:58751)
0929 10:26:15.226449 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.226497 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.227867 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.228489 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.228497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.228916 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:15.229476 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.229491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.230384 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.230387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.231306 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.231316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.231881 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.231884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.232035 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.232624 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:15.232640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:15.234277 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:15.236368 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:15.236374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:15.236391 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:15.236680 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:15.236990 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:15.236993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:15.236995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:15.237157 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:15.237627 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:15.237634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:15.237903 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.238119 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.238466 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":129,"threads_started":1}
I20260929 10:26:15.240615 4771 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46279
I20260929 10:26:15.240875 4771 heartbeater.cc:464] Registering TS with master...
I20260929 10:26:15.241423 4771 heartbeater.cc:511] Master 127.21.33.190:46279 requested a full tablet report, sending...
I20260929 10:26:15.242973 4580 ts_manager.cc:194] Registered new tserver with Master: b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079)
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:26:15.244159 4580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:58751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:26:15.264233 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46279
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:26:15.376439 4776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:15.376729 4776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:15.376773 4776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:15.381253 4776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:15.381333 4776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:15.381459 4776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:26:15.390547 4776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:15.391654 4776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:15.392681 4776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:15.402925 4784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:15.403036 4782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:15.402941 4781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:15.403221 4776 server_base.cc:1034] running on GCE node
I20260929 10:26:15.403715 4776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:15.404393 4776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:15.405584 4776 hybrid_clock.cc:648] HybridClock initialized: now 1790677575405566 us; error 31 us; skew 500 ppm
Sep 29 10:26:15 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677575, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:15.408629 4776 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:26:15.409663 4776 webserver.cc:460] Webserver started at http://127.21.33.130:41727/ using document root <none> and password file <none>
I20260929 10:26:15.410259 4776 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:15.410326 4776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:15.410580 4776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:26:15.413012 4776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "fc11f8ec6e2846a1866cb832daa7d43e"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "06f9dce6ca37662afd58bcc3b426e02e"
server_key_iv: "5ec5a8feecba8861c78f2bb66309c3bb"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.413771 4776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "fc11f8ec6e2846a1866cb832daa7d43e"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "06f9dce6ca37662afd58bcc3b426e02e"
server_key_iv: "5ec5a8feecba8861c78f2bb66309c3bb"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.418059 4776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:26:15.420857 4791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.422272 4776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:26:15.422430 4776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "fc11f8ec6e2846a1866cb832daa7d43e"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "06f9dce6ca37662afd58bcc3b426e02e"
server_key_iv: "5ec5a8feecba8861c78f2bb66309c3bb"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.422561 4776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:15.432099 4776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:15.435320 4776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:15.435540 4776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:15.436159 4776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:15.437001 4776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:26:15.437080 4776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.437160 4776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:26:15.437253 4776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 10:26:15.446133 4776 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:41489
I20260929 10:26:15.446159 4904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:41489 every 8 connection(s)
I20260929 10:26:15.447067 4776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:26:15.450935 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 4776
I20260929 10:26:15.451094 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:26:15.451392 21638 external_mini_cluster.cc:1428] Setting key 2cd3f6cce01d4c00d77296e99e0cca04
Sep 29 10:26:15 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677575, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:26:15.462203 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.448985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:38721 (local address 127.21.33.190:46279)
0929 10:26:15.449167 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.449170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.450130 (+ 960us) server_negotiation.cc:407] Connection header received
0929 10:26:15.451613 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.451616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.451674 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.451754 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:15.452915 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.453518 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.455134 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.455358 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.455492 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.458218 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:15.458225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:15.458230 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:15.458265 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:15.459965 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.460474 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.460478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.460480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.460577 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.460979 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.460984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.460989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.461271 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:15.461369 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.461969 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.462082 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 10:26:15.463099 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.449469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:38721)
0929 10:26:15.449934 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.449978 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.451351 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.451928 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.451935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.452293 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:15.452767 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.452780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.453654 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.453659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.454984 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.454998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.455543 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.455547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.455703 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.456341 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:15.456357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:15.458037 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:15.460095 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:15.460100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:15.460112 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:15.460368 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:15.460720 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:15.460723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:15.460725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:15.460863 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:15.461381 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:15.461387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:15.461728 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.461893 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.462156 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:26:15.464497 4905 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46279
I20260929 10:26:15.464756 4905 heartbeater.cc:464] Registering TS with master...
I20260929 10:26:15.465318 4905 heartbeater.cc:511] Master 127.21.33.190:46279 requested a full tablet report, sending...
I20260929 10:26:15.466434 4580 ts_manager.cc:194] Registered new tserver with Master: fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
I20260929 10:26:15.467015 4580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:38721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:26:15.481535 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46279
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:26:15.615592 4910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:15.615893 4910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:15.615983 4910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:15.620164 4910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:15.620245 4910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:15.620328 4910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:26:15.625916 4910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:15.627054 4910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:15.628083 4910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:15.637714 4918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:15.637714 4916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:15.637722 4915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:15.637964 4910 server_base.cc:1034] running on GCE node
I20260929 10:26:15.638453 4910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:15.639127 4910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:15.640298 4910 hybrid_clock.cc:648] HybridClock initialized: now 1790677575640265 us; error 44 us; skew 500 ppm
Sep 29 10:26:15 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677575, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:15.643194 4910 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:26:15.644045 4910 webserver.cc:460] Webserver started at http://127.21.33.131:39753/ using document root <none> and password file <none>
I20260929 10:26:15.644578 4910 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:15.644637 4910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:15.644800 4910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:26:15.646862 4910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "83e4820d9d4c4989a22c29ebbb3b85dc"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "f78c9baa69f0e4b1cadbd42bc814d355"
server_key_iv: "6fe890d726c8c7ff45eafab9aa856521"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.647395 4910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "83e4820d9d4c4989a22c29ebbb3b85dc"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "f78c9baa69f0e4b1cadbd42bc814d355"
server_key_iv: "6fe890d726c8c7ff45eafab9aa856521"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.651095 4910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 10:26:15.653594 4925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.654959 4910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:26:15.655124 4910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "83e4820d9d4c4989a22c29ebbb3b85dc"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "f78c9baa69f0e4b1cadbd42bc814d355"
server_key_iv: "6fe890d726c8c7ff45eafab9aa856521"
server_key_version: "encryptionkey@0"
I20260929 10:26:15.655249 4910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:15.712021 4910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:15.715583 4910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:15.715801 4910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:15.716434 4910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:15.717303 4910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:26:15.717377 4910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.717451 4910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:26:15.717500 4910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:15.727516 4910 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:39591
I20260929 10:26:15.727543 5038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:39591 every 8 connection(s)
I20260929 10:26:15.728442 4910 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:26:15.733443 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 4910
I20260929 10:26:15.733620 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:26:15.733928 21638 external_mini_cluster.cc:1428] Setting key dda6b18043dace9be0f1fe01e23ef97f
Sep 29 10:26:15 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677575, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:26:15.743479 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.730397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:32971 (local address 127.21.33.190:46279)
0929 10:26:15.730559 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.730562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.731407 (+ 845us) server_negotiation.cc:407] Connection header received
0929 10:26:15.732822 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.732825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.732888 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.733005 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:15.734530 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.735305 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.736835 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.737060 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.737153 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.739709 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:15.739712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:15.739714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:15.739739 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:15.741360 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.741900 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.741903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.741905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.741992 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.742371 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.742376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.742378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.742571 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:15.742682 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.743195 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.743317 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:26:15.744390 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.730733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:32971)
0929 10:26:15.731233 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.731281 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.732587 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.733249 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.733259 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.733735 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:15.734366 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.734388 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.735465 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.735473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.736703 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.736712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.737258 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.737264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.737426 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.738012 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:15.738028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:15.739534 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:15.741485 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:15.741491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:15.741507 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:15.741789 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:15.742108 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:15.742111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:15.742113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:15.742264 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:15.742681 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:15.742690 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:15.742947 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.743143 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.743480 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260929 10:26:15.745729 5039 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46279
I20260929 10:26:15.745963 5039 heartbeater.cc:464] Registering TS with master...
I20260929 10:26:15.746457 5039 heartbeater.cc:511] Master 127.21.33.190:46279 requested a full tablet report, sending...
I20260929 10:26:15.747553 4580 ts_manager.cc:194] Registered new tserver with Master: 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591)
I20260929 10:26:15.748329 4580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:32971
I20260929 10:26:15.748534 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:26:15.757952 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.750691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.21.33.190:46279)
0929 10:26:15.750841 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.750845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.751062 (+ 217us) server_negotiation.cc:407] Connection header received
0929 10:26:15.751107 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.751109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.751149 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.751233 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:15.751929 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.752467 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.753319 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.753534 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.753648 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.754921 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:15.754924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:15.754926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:15.754948 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:15.756230 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.756677 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.756679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.756681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.756752 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:15.757033 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:15.757035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:15.757036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:15.757329 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:15.757430 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.757618 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.757716 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:26:15.760965 4580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:26:15.764048 4580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:26:15.779443 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.772894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:43079 (local address 127.0.0.1:44972)
0929 10:26:15.773897 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.773912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.773992 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.774835 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.774841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.774863 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:15.775036 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.775044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.776577 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.776583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.777956 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.777967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.778852 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.778856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.778976 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.779015 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.779145 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.779285 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":60,"thread_start_us":99,"threads_started":1}
I20260929 10:26:15.779888 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.773894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.21.33.130:41489)
0929 10:26:15.774514 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.774522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.774541 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:26:15.774601 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.774604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.774822 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.774972 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:15.775395 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.776519 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.777861 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.778769 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.778901 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.779107 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.779612 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.779716 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":135,"threads_started":1}
I20260929 10:26:15.780035 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.772913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.21.33.131:39591)
0929 10:26:15.774274 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.774289 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.774316 (+ 27us) server_negotiation.cc:407] Connection header received
0929 10:26:15.774402 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.774409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.774649 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.774814 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:15.775304 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.776058 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.778103 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.778936 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.779069 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.779200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.779774 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.779868 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":143,"threads_started":1}
I20260929 10:26:15.780345 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.773372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.21.33.129:43079)
0929 10:26:15.774361 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.774368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.774389 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:26:15.774456 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.774460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.774649 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.774814 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:15.775183 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.776410 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.778118 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.778678 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.778841 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.779038 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.779318 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.779413 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":137,"threads_started":1}
I20260929 10:26:15.780359 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.772749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:39591 (local address 127.0.0.1:47250)
0929 10:26:15.773736 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.773792 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.773938 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.775010 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.775013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.775029 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:15.775161 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.775169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.776366 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.776370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.777961 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.777967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.779119 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.779123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.779224 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.779306 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.779387 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.779479 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":107,"thread_start_us":80,"threads_started":1}
I20260929 10:26:15.780476 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.773560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:41489 (local address 127.0.0.1:35044)
0929 10:26:15.774245 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.774257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.774320 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.775130 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.775133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.775152 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:15.775263 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.775269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.776665 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.776669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.777718 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.777728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.779182 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.779186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.779379 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.779406 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.779473 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.779532 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":78,"thread_start_us":115,"threads_started":1}
I20260929 10:26:15.783113 4705 tablet_service.cc:1467] Processing CreateTablet for tablet db85b80539f64cf685e30f7912cffafe (DEFAULT_TABLE table=test-table [id=79988927450448cc9ba0f8cab050eb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:26:15.783210 4973 tablet_service.cc:1467] Processing CreateTablet for tablet db85b80539f64cf685e30f7912cffafe (DEFAULT_TABLE table=test-table [id=79988927450448cc9ba0f8cab050eb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:26:15.783717 4839 tablet_service.cc:1467] Processing CreateTablet for tablet db85b80539f64cf685e30f7912cffafe (DEFAULT_TABLE table=test-table [id=79988927450448cc9ba0f8cab050eb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:26:15.784302 4705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db85b80539f64cf685e30f7912cffafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:26:15.784307 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db85b80539f64cf685e30f7912cffafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:26:15.784658 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db85b80539f64cf685e30f7912cffafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:26:15.790411 5056 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap starting.
I20260929 10:26:15.792155 5058 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Bootstrap starting.
I20260929 10:26:15.792739 5056 tablet_bootstrap.cc:654] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Neither blocks nor log segments found. Creating new log.
I20260929 10:26:15.793555 5056 log.cc:824] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:15.793570 5057 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Bootstrap starting.
I20260929 10:26:15.794811 5058 tablet_bootstrap.cc:654] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Neither blocks nor log segments found. Creating new log.
I20260929 10:26:15.795672 5058 log.cc:824] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:15.795742 5056 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: No bootstrap required, opened a new log
I20260929 10:26:15.795949 5056 ts_tablet_manager.cc:1397] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:26:15.796346 5057 tablet_bootstrap.cc:654] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Neither blocks nor log segments found. Creating new log.
I20260929 10:26:15.797188 5057 log.cc:824] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:15.797818 5058 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: No bootstrap required, opened a new log
I20260929 10:26:15.798079 5058 ts_tablet_manager.cc:1397] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:26:15.799532 5057 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: No bootstrap required, opened a new log
I20260929 10:26:15.799721 5057 ts_tablet_manager.cc:1397] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:26:15.800128 5056 raft_consensus.cc:348] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.800369 5056 raft_consensus.cc:374] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:26:15.800441 5056 raft_consensus.cc:729] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f4dc09ed9a42d2a00aa6265f9e807f, State: Initialized, Role: FOLLOWER
I20260929 10:26:15.800915 5056 consensus_queue.cc:260] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.802027 5056 ts_tablet_manager.cc:1428] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:26:15.802417 4771 heartbeater.cc:503] Master 127.21.33.190:46279 was elected leader, sending a full tablet report...
I20260929 10:26:15.802306 5058 raft_consensus.cc:348] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.802582 5058 raft_consensus.cc:374] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:26:15.802635 5058 raft_consensus.cc:729] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc11f8ec6e2846a1866cb832daa7d43e, State: Initialized, Role: FOLLOWER
I20260929 10:26:15.803078 5058 consensus_queue.cc:260] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.803628 5057 raft_consensus.cc:348] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.803913 4905 heartbeater.cc:503] Master 127.21.33.190:46279 was elected leader, sending a full tablet report...
I20260929 10:26:15.804037 5057 raft_consensus.cc:374] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:26:15.804105 5058 ts_tablet_manager.cc:1428] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:26:15.804098 5057 raft_consensus.cc:729] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e4820d9d4c4989a22c29ebbb3b85dc, State: Initialized, Role: FOLLOWER
I20260929 10:26:15.804525 5057 consensus_queue.cc:260] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.805249 5039 heartbeater.cc:503] Master 127.21.33.190:46279 was elected leader, sending a full tablet report...
I20260929 10:26:15.805614 5057 ts_tablet_manager.cc:1428] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:26:15.824851 5064 raft_consensus.cc:482] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:26:15.825114 5064 raft_consensus.cc:504] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.826436 5064 leader_election.cc:288] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079), fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
I20260929 10:26:15.831439 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.827044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:58487 (local address 127.21.33.130:41489)
0929 10:26:15.827243 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.827247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.827502 (+ 255us) server_negotiation.cc:407] Connection header received
0929 10:26:15.827592 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.827594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.827667 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.827749 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:15.827998 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.828676 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.830095 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.830714 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.830839 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.830890 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.831240 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.831311 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 10:26:15.831439 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.826929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:40343 (local address 127.21.33.129:43079)
0929 10:26:15.827119 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.827124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.827141 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:26:15.827199 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.827205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.827261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.827339 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:15.827667 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.828474 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.830095 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.830723 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.830839 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.830890 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.831240 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.831311 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:26:15.831459 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.826808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:43079 (local address 127.21.33.131:40343)
0929 10:26:15.826913 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.826945 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.827044 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.827385 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.827388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.827410 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:15.827550 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.827557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.828589 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.828591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.829943 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.829954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.830845 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.830848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.831026 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.831123 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.831205 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.831304 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":109}
I20260929 10:26:15.831473 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.826808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:41489 (local address 127.21.33.131:58487)
0929 10:26:15.827398 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.827410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:15.827502 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:15.827773 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:15.827775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:15.827790 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:15.827880 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.827884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.828778 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.828781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.829943 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:15.829954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.830845 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.830848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:15.831011 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.831127 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 10:26:15.831205 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 10:26:15.831304 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":207,"thread_start_us":144,"threads_started":1}
I20260929 10:26:15.832275 4859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85b80539f64cf685e30f7912cffafe" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" is_pre_election: true
I20260929 10:26:15.832274 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85b80539f64cf685e30f7912cffafe" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" is_pre_election: true
I20260929 10:26:15.832593 4859 raft_consensus.cc:2454] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 0.
I20260929 10:26:15.832593 4725 raft_consensus.cc:2454] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 0.
I20260929 10:26:15.833153 4927 leader_election.cc:302] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83e4820d9d4c4989a22c29ebbb3b85dc, b5f4dc09ed9a42d2a00aa6265f9e807f, fc11f8ec6e2846a1866cb832daa7d43e; no voters:
I20260929 10:26:15.833480 5064 raft_consensus.cc:2781] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:26:15.833597 5064 raft_consensus.cc:482] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:26:15.833657 5064 raft_consensus.cc:3037] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:26:15.834800 5064 raft_consensus.cc:504] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.835155 5064 leader_election.cc:288] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 election: Requested vote from peers b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079), fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
I20260929 10:26:15.835525 4859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85b80539f64cf685e30f7912cffafe" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc11f8ec6e2846a1866cb832daa7d43e"
I20260929 10:26:15.835525 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db85b80539f64cf685e30f7912cffafe" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f"
I20260929 10:26:15.835656 4859 raft_consensus.cc:3037] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:26:15.835656 4725 raft_consensus.cc:3037] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:26:15.836915 4859 raft_consensus.cc:2454] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 1.
I20260929 10:26:15.836915 4725 raft_consensus.cc:2454] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 1.
I20260929 10:26:15.837260 4927 leader_election.cc:302] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83e4820d9d4c4989a22c29ebbb3b85dc, b5f4dc09ed9a42d2a00aa6265f9e807f, fc11f8ec6e2846a1866cb832daa7d43e; no voters:
I20260929 10:26:15.837406 5064 raft_consensus.cc:2781] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:26:15.837630 5064 raft_consensus.cc:686] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 LEADER]: Becoming Leader. State: Replica: 83e4820d9d4c4989a22c29ebbb3b85dc, State: Running, Role: LEADER
I20260929 10:26:15.837960 5064 consensus_queue.cc:237] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:15.841351 4579 catalog_manager.cc:5495] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131). New cstate: current_term: 1 leader_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } health_report { overall_health: UNKNOWN } } }
I20260929 10:26:15.857177 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.853930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.21.33.131:39591)
0929 10:26:15.854072 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:15.854075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:15.854089 (+ 14us) server_negotiation.cc:407] Connection header received
0929 10:26:15.854160 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:15.854162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:15.854214 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:15.854302 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:26:15.854546 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.855034 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.855851 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:15.856072 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:15.856153 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:15.856419 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:26:15.856850 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:26:15.856938 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:15.857000 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 10:26:15.857053 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:26:15.863458 4725 raft_consensus.cc:1261] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Refusing update from remote peer 83e4820d9d4c4989a22c29ebbb3b85dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:26:15.863458 4859 raft_consensus.cc:1261] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Refusing update from remote peer 83e4820d9d4c4989a22c29ebbb3b85dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:26:15.864308 5064 consensus_queue.cc:1035] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:26:15.864567 5066 consensus_queue.cc:1035] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:26:15.873898 5071 mvcc.cc:205] Tried to move back new op lower bound from 7334615350728568832 to 7334615350635220992. Current Snapshot: MvccSnapshot[applied={T|T < 7334615350728568832}]
I20260929 10:26:15.881318 4580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:26:15.881500 4580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:26:15.883965 4580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Sending DeleteTablet for 3 replicas of tablet db85b80539f64cf685e30f7912cffafe
I20260929 10:26:15.884763 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet db85b80539f64cf685e30f7912cffafe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:26:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:47250
I20260929 10:26:15.884968 4839 tablet_service.cc:1514] Processing DeleteTablet for tablet db85b80539f64cf685e30f7912cffafe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:26:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:35044
I20260929 10:26:15.885142 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 4642
I20260929 10:26:15.886247 5080 tablet_replica.cc:330] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: stopping tablet replica
I20260929 10:26:15.886596 5080 raft_consensus.cc:2229] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:26:15.886936 5080 raft_consensus.cc:2258] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:26:15.887835 5079 tablet_replica.cc:330] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: stopping tablet replica
I20260929 10:26:15.888140 5079 raft_consensus.cc:2229] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:26:15.888507 5079 raft_consensus.cc:2258] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:26:15.888710 5080 ts_tablet_manager.cc:1905] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:26:15.891036 5079 ts_tablet_manager.cc:1905] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:26:15.891546 5080 ts_tablet_manager.cc:1918] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:26:15.891754 5080 log.cc:1196] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/db85b80539f64cf685e30f7912cffafe
W20260929 10:26:15.891886 4567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:43079 (error 108)
I20260929 10:26:15.892217 5080 ts_tablet_manager.cc:1939] T db85b80539f64cf685e30f7912cffafe P fc11f8ec6e2846a1866cb832daa7d43e: Deleting consensus metadata
I20260929 10:26:15.892870 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 4776
I20260929 10:26:15.894883 5079 ts_tablet_manager.cc:1918] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:26:15.894989 5079 log.cc:1196] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/db85b80539f64cf685e30f7912cffafe
I20260929 10:26:15.895372 5079 ts_tablet_manager.cc:1939] T db85b80539f64cf685e30f7912cffafe P 83e4820d9d4c4989a22c29ebbb3b85dc: Deleting consensus metadata
I20260929 10:26:15.896673 4566 catalog_manager.cc:4841] TS 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591): tablet db85b80539f64cf685e30f7912cffafe (table test-table [id=79988927450448cc9ba0f8cab050eb0d]) successfully deleted
I20260929 10:26:15.897341 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.896980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:43079 (local address 127.0.0.1:44976)
0929 10:26:15.897123 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.897230 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 10:26:15.897523 4567 catalog_manager.cc:4584] TS b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db85b80539f64cf685e30f7912cffafe: Network error: Client connection negotiation failed: client connection to 127.21.33.129:43079: connect: Connection refused (error 111)
I20260929 10:26:15.899653 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 4910
I20260929 10:26:15.901376 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:15.899831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:41489 (local address 127.0.0.1:35056)
0929 10:26:15.901162 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:15.901285 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.130:41489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1251}
I20260929 10:26:15.905951 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 4548
I20260929 10:26:35.912948 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46279
--webserver_interface=127.21.33.190
--webserver_port=46719
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:46279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:26:36.023959 5082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:36.024238 5082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:36.024279 5082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:36.028357 5082 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:26:36.028416 5082 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:26:36.028434 5082 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:26:36.028447 5082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:26:36.028465 5082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:36.028517 5082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:36.028545 5082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:26:36.028573 5082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:26:36.033927 5082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:46279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=46719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:36.035027 5082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:36.036075 5082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:36.049445 5089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.049552 5088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.049525 5091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:36.049715 5082 server_base.cc:1034] running on GCE node
I20260929 10:26:36.050426 5082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:36.051589 5082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:36.052780 5082 hybrid_clock.cc:648] HybridClock initialized: now 1790677596052750 us; error 56 us; skew 500 ppm
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:36.055617 5082 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:26:36.056559 5082 webserver.cc:460] Webserver started at http://127.21.33.190:46719/ using document root <none> and password file <none>
I20260929 10:26:36.057179 5082 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:36.057341 5082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:36.061146 5082 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 10:26:36.063373 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.064683 5082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:26:36.064844 5082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "3e98fefa941c4c2a9aadb17d3c04b396"
format_stamp: "Formatted at 2026-09-29 10:26:14 on dist-test-slave-2rbl"
server_key: "ef85f814435eec64856cc1d2eef67227"
server_key_iv: "5ae54d52dfa2f0e14552f825e5cc4685"
server_key_version: "encryptionkey@0"
I20260929 10:26:36.065523 5082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:36.076701 5082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:36.079736 5082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:36.079939 5082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:36.087019 5082 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:46279
I20260929 10:26:36.087301 5150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:46279 every 8 connection(s)
I20260929 10:26:36.088080 5082 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:26:36.089823 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5082
I20260929 10:26:36.090344 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:43079
--local_ip_for_outbound_sockets=127.21.33.129
--tserver_master_addrs=127.21.33.190:46279
--webserver_port=34021
--webserver_interface=127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:26:36.092999 5151 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:26:36.094774 5151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Bootstrap starting.
I20260929 10:26:36.111799 5151 log.cc:824] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:36.120359 5151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:26:36.120766 5151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Bootstrap complete.
I20260929 10:26:36.124868 5151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:36.125313 5151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e98fefa941c4c2a9aadb17d3c04b396, State: Initialized, Role: FOLLOWER
I20260929 10:26:36.125803 5151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:36.125913 5151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:26:36.126003 5151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:26:36.126120 5151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 1 FOLLOWER]: Advancing to term 2
I20260929 10:26:36.128013 5151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:36.128345 5151 leader_election.cc:302] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e98fefa941c4c2a9aadb17d3c04b396; no voters:
I20260929 10:26:36.128651 5151 leader_election.cc:288] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 10:26:36.128785 5155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 10:26:36.129027 5155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 LEADER]: Becoming Leader. State: Replica: 3e98fefa941c4c2a9aadb17d3c04b396, State: Running, Role: LEADER
I20260929 10:26:36.129400 5155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:36.129856 5151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:26:36.130782 5156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } } }
I20260929 10:26:36.130918 5156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: This master's current role is: LEADER
I20260929 10:26:36.131497 5157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e98fefa941c4c2a9aadb17d3c04b396. Latest consensus state: current_term: 2 leader_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } } }
I20260929 10:26:36.131672 5157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: This master's current role is: LEADER
I20260929 10:26:36.131960 5164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:26:36.137343 5164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:26:36.137914 5164 catalog_manager.cc:1259] Loaded cluster ID: 7b6290d5cf204bbf9878b4d23a061820
I20260929 10:26:36.137972 5164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:26:36.140355 5164 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:26:36.141016 5164 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Loaded TSK: 0
I20260929 10:26:36.155493 5164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Generated new TSK 1
I20260929 10:26:36.155777 5164 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:26:36.209041 5153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:36.209368 5153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:36.209443 5153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:36.213968 5153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:36.214033 5153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:36.214124 5153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:26:36.219501 5153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:43079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=34021
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:36.220532 5153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:36.221675 5153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:36.231791 5181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.231799 5179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.231775 5178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:36.232489 5153 server_base.cc:1034] running on GCE node
I20260929 10:26:36.232894 5153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:36.233687 5153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:36.234861 5153 hybrid_clock.cc:648] HybridClock initialized: now 1790677596234839 us; error 40 us; skew 500 ppm
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:36.237970 5153 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:26:36.239110 5153 webserver.cc:460] Webserver started at http://127.21.33.129:34021/ using document root <none> and password file <none>
I20260929 10:26:36.239712 5153 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:36.239833 5153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:36.244014 5153 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:26:36.246510 5188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.247860 5153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 10:26:36.247960 5153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "32a87e189d25d7785d78ecb966e9e919"
server_key_iv: "b9f6299bca94afb9e227ace3c50b51d1"
server_key_version: "encryptionkey@0"
I20260929 10:26:36.248606 5153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:36.291980 5153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:36.295468 5153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:36.295689 5153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:36.296321 5153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:36.297530 5195 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:26:36.301709 5153 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:26:36.301786 5153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 10:26:36.301862 5153 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:26:36.303639 5153 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:26:36.303685 5153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 10:26:36.304020 5195 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap starting.
I20260929 10:26:36.314121 5153 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:43079
I20260929 10:26:36.314220 5302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:43079 every 8 connection(s)
I20260929 10:26:36.315119 5153 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:26:36.317814 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5153
I20260929 10:26:36.318279 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:41489
--local_ip_for_outbound_sockets=127.21.33.130
--tserver_master_addrs=127.21.33.190:46279
--webserver_port=41727
--webserver_interface=127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:26:36.323773 5195 log.cc:824] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:26:36.332228 5195 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:26:36.332666 5195 tablet_bootstrap.cc:492] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap complete.
I20260929 10:26:36.334138 5195 ts_tablet_manager.cc:1397] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.018s
I20260929 10:26:36.335158 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.317078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:38331)
0929 10:26:36.318628 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:36.318674 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:36.319491 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:36.320875 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:36.320884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:36.321459 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:36.322173 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.322195 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.323549 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.323553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.324668 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.324678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.325925 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.325929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.326130 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.326862 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:36.326882 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:36.328975 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:36.331694 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.331699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.331711 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.332033 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.332430 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.332435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.332438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.332624 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.333423 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:36.333432 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:36.333743 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 10:26:36.333929 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 10:26:36.334242 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":173,"threads_started":1}
I20260929 10:26:36.335522 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.317260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:38331 (local address 127.21.33.190:46279)
0929 10:26:36.317719 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.317738 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.318808 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 10:26:36.319945 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.319963 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.320395 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.320779 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:36.322380 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.323383 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.324839 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.325756 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.325966 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.329227 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:36.329234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:36.329250 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:36.329284 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:36.331547 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.332164 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.332169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.332174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.332319 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.332744 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.332747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.332749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.333243 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:36.333369 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.334434 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.334746 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":113,"threads_started":1}
I20260929 10:26:36.337021 5195 raft_consensus.cc:348] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:36.337550 5195 raft_consensus.cc:729] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f4dc09ed9a42d2a00aa6265f9e807f, State: Initialized, Role: FOLLOWER
I20260929 10:26:36.338104 5195 consensus_queue.cc:260] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } }
I20260929 10:26:36.338418 5303 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46279
I20260929 10:26:36.338630 5303 heartbeater.cc:464] Registering TS with master...
I20260929 10:26:36.339120 5195 ts_tablet_manager.cc:1428] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 10:26:36.339146 5303 heartbeater.cc:511] Master 127.21.33.190:46279 requested a full tablet report, sending...
I20260929 10:26:36.341758 5115 ts_manager.cc:194] Registered new tserver with Master: b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079)
I20260929 10:26:36.345045 5115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:38331
I20260929 10:26:36.347759 5303 heartbeater.cc:503] Master 127.21.33.190:46279 was elected leader, sending a full tablet report...
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.129@KRBTEST.COM
I20260929 10:26:36.355173 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.344482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:43079 (local address 127.0.0.1:45754)
0929 10:26:36.345267 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:36.345310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:36.345501 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:36.346032 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:36.346038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:36.346414 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:36.346514 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.346520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.347485 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.347488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.348255 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.348264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.348991 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.348994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.349099 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.349768 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:36.349780 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:36.351312 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:36.353514 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.353522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.353526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.353771 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.354066 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.354068 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.354069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.354150 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.354649 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:36.354654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:36.354756 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 10:26:36.354870 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 10:26:36.355017 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":129,"threads_started":1}
I20260929 10:26:36.355295 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.344599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.21.33.129:43079)
0929 10:26:36.344950 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.344961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.345403 (+ 442us) server_negotiation.cc:407] Connection header received
0929 10:26:36.345672 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.345678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.345877 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.346012 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:36.346644 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.347357 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.348370 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.348855 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.348966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.351454 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:36.351464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:36.351476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:36.351502 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:36.353409 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.353877 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.353883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.353885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.353977 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.354233 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.354236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.354238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.354517 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:36.354657 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.354970 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.355139 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260929 10:26:36.356387 5237 tablet_service.cc:1514] Processing DeleteTablet for tablet db85b80539f64cf685e30f7912cffafe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:26:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:45754
I20260929 10:26:36.357401 5312 tablet_replica.cc:330] stopping tablet replica
I20260929 10:26:36.357700 5312 raft_consensus.cc:2229] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:26:36.357949 5312 raft_consensus.cc:2258] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:26:36.358918 5312 ts_tablet_manager.cc:1905] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:26:36.362593 5312 ts_tablet_manager.cc:1918] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:26:36.362706 5312 log.cc:1196] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal/wals/db85b80539f64cf685e30f7912cffafe
I20260929 10:26:36.362980 5312 ts_tablet_manager.cc:1939] T db85b80539f64cf685e30f7912cffafe P b5f4dc09ed9a42d2a00aa6265f9e807f: Deleting consensus metadata
I20260929 10:26:36.364760 5102 catalog_manager.cc:4841] TS b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079): tablet db85b80539f64cf685e30f7912cffafe (table test-table [id=79988927450448cc9ba0f8cab050eb0d]) successfully deleted
W20260929 10:26:36.440733 5307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:36.440999 5307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:36.441042 5307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:36.445127 5307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:36.445242 5307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:36.445397 5307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:26:36.450913 5307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:41489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=41727
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:36.452062 5307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:36.453163 5307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:36.462836 5318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.462836 5320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.462889 5317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:36.463054 5307 server_base.cc:1034] running on GCE node
I20260929 10:26:36.463689 5307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:36.464339 5307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:36.465538 5307 hybrid_clock.cc:648] HybridClock initialized: now 1790677596465530 us; error 37 us; skew 500 ppm
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:36.468305 5307 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:26:36.469306 5307 webserver.cc:460] Webserver started at http://127.21.33.130:41727/ using document root <none> and password file <none>
I20260929 10:26:36.469902 5307 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:36.469980 5307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:36.474254 5307 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:26:36.476341 5327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.477711 5307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:26:36.477824 5307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "fc11f8ec6e2846a1866cb832daa7d43e"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "06f9dce6ca37662afd58bcc3b426e02e"
server_key_iv: "5ec5a8feecba8861c78f2bb66309c3bb"
server_key_version: "encryptionkey@0"
I20260929 10:26:36.478379 5307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:36.492873 5307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:36.495913 5307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:36.496119 5307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:36.496706 5307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:36.497539 5307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:26:36.497584 5307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.497648 5307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:26:36.497684 5307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.507256 5307 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:41489
I20260929 10:26:36.507270 5440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:41489 every 8 connection(s)
I20260929 10:26:36.508299 5307 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:26:36.518350 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5307
I20260929 10:26:36.518846 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:39591
--local_ip_for_outbound_sockets=127.21.33.131
--tserver_master_addrs=127.21.33.190:46279
--webserver_port=39753
--webserver_interface=127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:26:36.521663 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.510129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:32937 (local address 127.21.33.190:46279)
0929 10:26:36.510304 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.510309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.511129 (+ 820us) server_negotiation.cc:407] Connection header received
0929 10:26:36.512422 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.512425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.512475 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.512549 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:36.513752 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.514332 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.515409 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.515629 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.515703 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.518116 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:36.518119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:36.518121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:36.518144 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:36.519616 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.520124 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.520130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.520132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.520209 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.520545 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.520550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.520551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.520736 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:36.520841 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.521359 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.521491 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 10:26:36.522459 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.510523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:32937)
0929 10:26:36.510969 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:36.511010 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:36.512224 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:36.512719 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:36.512726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:36.513118 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:36.513608 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.513620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.514442 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.514444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.515309 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.515318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.515751 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.515753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.515893 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.516444 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:36.516460 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:36.517949 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:36.519736 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.519744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.519760 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.520018 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.520306 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.520309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.520311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.520449 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.520839 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:36.520846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:36.521081 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 10:26:36.521316 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 10:26:36.521633 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20260929 10:26:36.523658 5441 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46279
I20260929 10:26:36.523912 5441 heartbeater.cc:464] Registering TS with master...
I20260929 10:26:36.524377 5441 heartbeater.cc:511] Master 127.21.33.190:46279 requested a full tablet report, sending...
I20260929 10:26:36.525457 5115 ts_manager.cc:194] Registered new tserver with Master: fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
I20260929 10:26:36.526341 5115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:32937
W20260929 10:26:36.631480 5444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:36.631762 5444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:36.631805 5444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:36.636127 5444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:36.636186 5444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:36.636281 5444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:26:36.641721 5444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:39591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=39753
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:36.642848 5444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:36.643918 5444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:36.654022 5449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.654093 5452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:36.654078 5450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:36.654438 5444 server_base.cc:1034] running on GCE node
I20260929 10:26:36.654882 5444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:36.655514 5444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:36.656736 5444 hybrid_clock.cc:648] HybridClock initialized: now 1790677596656720 us; error 25 us; skew 500 ppm
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:36.659704 5444 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:26:36.660594 5444 webserver.cc:460] Webserver started at http://127.21.33.131:39753/ using document root <none> and password file <none>
I20260929 10:26:36.661173 5444 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:36.661329 5444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:36.665269 5444 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:26:36.667286 5459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.668534 5444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:26:36.668630 5444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "83e4820d9d4c4989a22c29ebbb3b85dc"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "f78c9baa69f0e4b1cadbd42bc814d355"
server_key_iv: "6fe890d726c8c7ff45eafab9aa856521"
server_key_version: "encryptionkey@0"
I20260929 10:26:36.669369 5444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:36.681964 5444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:36.684902 5444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:36.685113 5444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:36.685760 5444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:36.686589 5444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:26:36.686640 5444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.686681 5444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:26:36.686741 5444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:36.695266 5444 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:39591
I20260929 10:26:36.695271 5572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:39591 every 8 connection(s)
I20260929 10:26:36.696285 5444 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:26:36.705590 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5444
Sep 29 10:26:36 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:26:36.710597 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.698267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:47039 (local address 127.21.33.190:46279)
0929 10:26:36.698416 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.698419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.699320 (+ 901us) server_negotiation.cc:407] Connection header received
0929 10:26:36.700679 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.700683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.700741 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.700849 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:36.702074 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.702587 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.703726 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.703953 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.704076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.706761 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:36.706764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:36.706766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:36.706788 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:36.708434 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.708939 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.708942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.708943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.709017 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.709436 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.709439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.709441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.709636 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:36.709724 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.710316 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.710411 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 10:26:36.711727 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.698669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:47039)
0929 10:26:36.699147 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:36.699191 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:36.700450 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:36.701028 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:36.701035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:36.701420 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:36.701911 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.701924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.702714 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.702719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.703598 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.703607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.704104 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.704109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.704279 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.704903 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:36.704920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:36.706586 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:36.708545 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.708550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.708562 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.708814 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.709107 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:36.709110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:36.709112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:36.709313 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:36.709752 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:36.709761 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:36.710030 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 10:26:36.710236 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 10:26:36.710588 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
I20260929 10:26:36.713065 5573 heartbeater.cc:347] Connected to a master server at 127.21.33.190:46279
I20260929 10:26:36.713378 5573 heartbeater.cc:464] Registering TS with master...
I20260929 10:26:36.713984 5573 heartbeater.cc:511] Master 127.21.33.190:46279 requested a full tablet report, sending...
I20260929 10:26:36.715162 5115 ts_manager.cc:194] Registered new tserver with Master: 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591)
I20260929 10:26:36.715379 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.706246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.21.33.190:46279)
0929 10:26:36.706532 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.706536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.707001 (+ 465us) server_negotiation.cc:407] Connection header received
0929 10:26:36.707030 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.707032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.707065 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.707136 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:36.708020 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.708781 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.709851 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.710068 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.710155 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.711752 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:36.711755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:36.711757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:36.711778 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:36.713390 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.713903 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.713905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.713906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.713976 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.714244 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.714246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.714247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.714405 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:36.714546 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.714795 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.714923 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260929 10:26:36.716018 5115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:47039
I20260929 10:26:36.717226 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:26:36.720990 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.717852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.21.33.190:46279)
0929 10:26:36.717976 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.717980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.718131 (+ 151us) server_negotiation.cc:407] Connection header received
0929 10:26:36.718242 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.718245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.718358 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.718472 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:26:36.718689 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.719177 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.719969 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.720196 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.720280 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.720599 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:26:36.720714 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 10:26:36.720905 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55238: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 10:26:36.721072 5576 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55238: token expired
W20260929 10:26:36.721503 5042 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.33.190:46279)
I20260929 10:26:36.729056 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.721898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.21.33.190:46279)
0929 10:26:36.722012 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.722016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.722056 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:26:36.722246 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.722249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.722293 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.722373 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:36.723011 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.723481 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.724304 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.724519 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.724649 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.726024 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:36.726030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:36.726038 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:36.726059 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:36.727405 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.727836 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.727840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.727842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.727923 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:36.728229 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:36.728232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:36.728234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:36.728403 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:36.728541 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.728776 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.728892 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:26:36.731781 5115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:26:36.735661 5115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:26:36.743845 5237 tablet_service.cc:1467] Processing CreateTablet for tablet 2583c878e1104935b41161c87eff6e02 (DEFAULT_TABLE table=test-table [id=e512c8ce70584fdfb8fa76d9ec6bc270]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:26:36.744545 5237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2583c878e1104935b41161c87eff6e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:26:36.747380 5195 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap starting.
I20260929 10:26:36.748322 5195 tablet_bootstrap.cc:654] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Neither blocks nor log segments found. Creating new log.
I20260929 10:26:36.748518 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.742580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:39591 (local address 127.0.0.1:50984)
0929 10:26:36.742748 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:36.742764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:36.742844 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:36.744014 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:36.744020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:36.744041 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:36.744147 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.744153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.745866 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.745870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.747060 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.747074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.748073 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.748076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.748173 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.748228 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 10:26:36.748283 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 10:26:36.748367 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 10:26:36.748883 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.743319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.21.33.130:41489)
0929 10:26:36.743728 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.743738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.743759 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:26:36.743827 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.743832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.744000 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.744149 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:36.744600 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.745837 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.747018 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.747887 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.748009 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.748194 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.748597 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.748715 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":111,"threads_started":1}
I20260929 10:26:36.748926 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.742918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:41489 (local address 127.0.0.1:44302)
0929 10:26:36.743363 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:36.743377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:36.743464 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:36.744356 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:36.744359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:36.744378 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:36.744465 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.744472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.746090 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.746093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.746882 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:36.746891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.748255 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.748261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:36.748361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.748380 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:26:36.748443 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 10:26:36.748496 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":106,"threads_started":1}
I20260929 10:26:36.749326 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:36.743036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.21.33.131:39591)
0929 10:26:36.743456 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:36.743468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:36.743496 (+ 28us) server_negotiation.cc:407] Connection header received
0929 10:26:36.743580 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:36.743587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:36.743842 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:36.744010 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:36.744276 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.745746 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.747198 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:36.747945 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:36.748818 (+ 873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:36.748978 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:36.749035 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 10:26:36.749145 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":195,"threads_started":1}
I20260929 10:26:36.750155 5195 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: No bootstrap required, opened a new log
I20260929 10:26:36.750238 5195 ts_tablet_manager.cc:1397] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 10:26:36.750707 5195 raft_consensus.cc:348] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:36.750833 5195 raft_consensus.cc:374] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:26:36.750869 5195 raft_consensus.cc:729] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f4dc09ed9a42d2a00aa6265f9e807f, State: Initialized, Role: FOLLOWER
I20260929 10:26:36.750980 5195 consensus_queue.cc:260] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:36.751216 5195 ts_tablet_manager.cc:1428] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 10:26:36.752300 5507 tablet_service.cc:1467] Processing CreateTablet for tablet 2583c878e1104935b41161c87eff6e02 (DEFAULT_TABLE table=test-table [id=e512c8ce70584fdfb8fa76d9ec6bc270]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:26:36.753389 5507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2583c878e1104935b41161c87eff6e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:26:36.755666 5375 tablet_service.cc:1467] Processing CreateTablet for tablet 2583c878e1104935b41161c87eff6e02 (DEFAULT_TABLE table=test-table [id=e512c8ce70584fdfb8fa76d9ec6bc270]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:26:36.756527 5375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2583c878e1104935b41161c87eff6e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:26:36.760608 5586 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Bootstrap starting.
I20260929 10:26:36.762591 5586 tablet_bootstrap.cc:654] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Neither blocks nor log segments found. Creating new log.
I20260929 10:26:36.763090 5587 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Bootstrap starting.
I20260929 10:26:36.763837 5586 log.cc:824] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:36.765549 5587 tablet_bootstrap.cc:654] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Neither blocks nor log segments found. Creating new log.
I20260929 10:26:36.765830 5586 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: No bootstrap required, opened a new log
I20260929 10:26:36.766057 5586 ts_tablet_manager.cc:1397] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:26:36.766598 5587 log.cc:824] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:36.768352 5587 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: No bootstrap required, opened a new log
I20260929 10:26:36.768559 5587 ts_tablet_manager.cc:1397] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 10:26:36.769381 5586 raft_consensus.cc:348] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:36.769681 5586 raft_consensus.cc:374] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:26:36.769769 5586 raft_consensus.cc:729] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e4820d9d4c4989a22c29ebbb3b85dc, State: Initialized, Role: FOLLOWER
I20260929 10:26:36.770223 5586 consensus_queue.cc:260] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:36.770905 5573 heartbeater.cc:503] Master 127.21.33.190:46279 was elected leader, sending a full tablet report...
I20260929 10:26:36.771165 5586 ts_tablet_manager.cc:1428] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:26:36.771878 5587 raft_consensus.cc:348] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:36.772120 5587 raft_consensus.cc:374] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:26:36.772187 5587 raft_consensus.cc:729] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc11f8ec6e2846a1866cb832daa7d43e, State: Initialized, Role: FOLLOWER
I20260929 10:26:36.772682 5587 consensus_queue.cc:260] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:36.773612 5587 ts_tablet_manager.cc:1428] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 10:26:36.773958 5441 heartbeater.cc:503] Master 127.21.33.190:46279 was elected leader, sending a full tablet report...
W20260929 10:26:36.816439 5304 tablet.cc:2369] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:26:36.947331 5574 tablet.cc:2369] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:26:37.009506 5442 tablet.cc:2369] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:26:37.216810 5590 raft_consensus.cc:482] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:26:37.217073 5590 raft_consensus.cc:504] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:37.218361 5590 leader_election.cc:288] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489), b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079)
I20260929 10:26:37.223613 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:37.218782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:52037 (local address 127.21.33.130:41489)
0929 10:26:37.218953 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:37.218958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:37.219093 (+ 135us) server_negotiation.cc:407] Connection header received
0929 10:26:37.219281 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:37.219284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:37.219332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:37.219436 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:37.219737 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.220519 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.222234 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.222831 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.222986 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:37.223056 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:37.223388 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 10:26:37.223476 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:26:37.223636 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:37.218784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:39287 (local address 127.21.33.129:43079)
0929 10:26:37.219130 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:37.219134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:37.219393 (+ 259us) server_negotiation.cc:407] Connection header received
0929 10:26:37.219527 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:37.219530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:37.219611 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:37.219729 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:26:37.219981 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.220708 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.222234 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.222921 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.223029 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:37.223179 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:37.223383 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 10:26:37.223484 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
I20260929 10:26:37.223615 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:37.218637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:41489 (local address 127.21.33.131:52037)
0929 10:26:37.218963 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:37.218982 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:37.219146 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:37.219474 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:37.219480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:37.219500 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:37.219624 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:37.219629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.220649 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.220652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:37.222085 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:37.222096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.223039 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.223044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:37.223219 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:37.223234 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 10:26:37.223324 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:26:37.223424 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":97,"thread_start_us":98,"threads_started":1}
I20260929 10:26:37.223626 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:37.218637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:43079 (local address 127.21.33.131:39287)
0929 10:26:37.219297 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:37.219309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:37.219395 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:37.219755 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:37.219758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:37.219783 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:26:37.219873 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:37.219878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.220816 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.220819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:37.222097 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:37.222103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.223039 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.223044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:37.223206 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:37.223234 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:26:37.223324 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 10:26:37.223424 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":405,"thread_start_us":67,"threads_started":1}
I20260929 10:26:37.224452 5257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2583c878e1104935b41161c87eff6e02" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" is_pre_election: true
I20260929 10:26:37.224546 5395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2583c878e1104935b41161c87eff6e02" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" is_pre_election: true
I20260929 10:26:37.224817 5257 raft_consensus.cc:2454] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 0.
I20260929 10:26:37.224885 5395 raft_consensus.cc:2454] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 0.
I20260929 10:26:37.225327 5463 leader_election.cc:302] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e4820d9d4c4989a22c29ebbb3b85dc, b5f4dc09ed9a42d2a00aa6265f9e807f; no voters:
I20260929 10:26:37.225625 5590 raft_consensus.cc:2781] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:26:37.225710 5590 raft_consensus.cc:482] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:26:37.225755 5590 raft_consensus.cc:3037] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:26:37.226778 5590 raft_consensus.cc:504] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:37.227103 5590 leader_election.cc:288] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 election: Requested vote from peers fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489), b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079)
I20260929 10:26:37.227411 5395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2583c878e1104935b41161c87eff6e02" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc11f8ec6e2846a1866cb832daa7d43e"
I20260929 10:26:37.227437 5257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2583c878e1104935b41161c87eff6e02" candidate_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f"
I20260929 10:26:37.227563 5257 raft_consensus.cc:3037] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:26:37.227552 5395 raft_consensus.cc:3037] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:26:37.228545 5257 raft_consensus.cc:2454] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 1.
I20260929 10:26:37.228674 5395 raft_consensus.cc:2454] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e4820d9d4c4989a22c29ebbb3b85dc in term 1.
I20260929 10:26:37.228996 5461 leader_election.cc:302] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e4820d9d4c4989a22c29ebbb3b85dc, fc11f8ec6e2846a1866cb832daa7d43e; no voters:
I20260929 10:26:37.229244 5590 raft_consensus.cc:2781] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:26:37.229475 5590 raft_consensus.cc:686] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 LEADER]: Becoming Leader. State: Replica: 83e4820d9d4c4989a22c29ebbb3b85dc, State: Running, Role: LEADER
I20260929 10:26:37.229827 5590 consensus_queue.cc:237] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:37.233006 5115 catalog_manager.cc:5495] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131). New cstate: current_term: 1 leader_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } health_report { overall_health: UNKNOWN } } }
I20260929 10:26:37.312139 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:37.308330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.21.33.131:39591)
0929 10:26:37.308635 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:37.308639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:37.308654 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:26:37.308794 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:37.308796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:37.308876 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:37.308996 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:26:37.309317 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.309812 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.310679 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:37.310904 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:37.310995 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:37.311307 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:26:37.311716 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:26:37.311807 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:37.311877 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 10:26:37.311922 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
I20260929 10:26:37.317942 5257 raft_consensus.cc:1261] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Refusing update from remote peer 83e4820d9d4c4989a22c29ebbb3b85dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:26:37.317946 5395 raft_consensus.cc:1261] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Refusing update from remote peer 83e4820d9d4c4989a22c29ebbb3b85dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:26:37.318641 5595 consensus_queue.cc:1035] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:26:37.318903 5590 consensus_queue.cc:1035] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:26:37.325567 5599 mvcc.cc:205] Tried to move back new op lower bound from 7334615438606983168 to 7334615438256394240. Current Snapshot: MvccSnapshot[applied={T|T < 7334615438606983168}]
I20260929 10:26:37.327337 5603 mvcc.cc:205] Tried to move back new op lower bound from 7334615438606983168 to 7334615438256394240. Current Snapshot: MvccSnapshot[applied={T|T < 7334615438606983168}]
I20260929 10:26:37.328220 5602 mvcc.cc:205] Tried to move back new op lower bound from 7334615438606983168 to 7334615438256394240. Current Snapshot: MvccSnapshot[applied={T|T < 7334615438606983168}]
I20260929 10:26:37.335003 5114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:26:37.335182 5114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:26:37.337563 5114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Sending DeleteTablet for 3 replicas of tablet 2583c878e1104935b41161c87eff6e02
I20260929 10:26:37.338276 5237 tablet_service.cc:1514] Processing DeleteTablet for tablet 2583c878e1104935b41161c87eff6e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:26:37 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:45754
I20260929 10:26:37.338284 5507 tablet_service.cc:1514] Processing DeleteTablet for tablet 2583c878e1104935b41161c87eff6e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:26:37 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:50984
I20260929 10:26:37.338299 5375 tablet_service.cc:1514] Processing DeleteTablet for tablet 2583c878e1104935b41161c87eff6e02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:26:37 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:44302
I20260929 10:26:37.338657 5611 tablet_replica.cc:330] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: stopping tablet replica
I20260929 10:26:37.338814 5611 raft_consensus.cc:2229] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:26:37.339092 5613 tablet_replica.cc:330] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: stopping tablet replica
I20260929 10:26:37.339103 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 5153
I20260929 10:26:37.339409 5613 raft_consensus.cc:2229] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:26:37.339725 5613 raft_consensus.cc:2258] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:26:37.340106 5612 tablet_replica.cc:330] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: stopping tablet replica
I20260929 10:26:37.340394 5612 raft_consensus.cc:2229] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:26:37.340788 5612 raft_consensus.cc:2258] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:26:37.341217 5613 ts_tablet_manager.cc:1905] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:26:37.342038 5612 ts_tablet_manager.cc:1905] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:26:37.343977 5613 ts_tablet_manager.cc:1918] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:26:37.344061 5613 log.cc:1196] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/2583c878e1104935b41161c87eff6e02
I20260929 10:26:37.344369 5613 ts_tablet_manager.cc:1939] T 2583c878e1104935b41161c87eff6e02 P fc11f8ec6e2846a1866cb832daa7d43e: Deleting consensus metadata
I20260929 10:26:37.345436 5612 ts_tablet_manager.cc:1918] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:26:37.345531 5612 log.cc:1196] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/2583c878e1104935b41161c87eff6e02
I20260929 10:26:37.345850 5612 ts_tablet_manager.cc:1939] T 2583c878e1104935b41161c87eff6e02 P 83e4820d9d4c4989a22c29ebbb3b85dc: Deleting consensus metadata
I20260929 10:26:37.346483 5100 catalog_manager.cc:4841] TS fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489): tablet 2583c878e1104935b41161c87eff6e02 (table test-table [id=e512c8ce70584fdfb8fa76d9ec6bc270]) successfully deleted
I20260929 10:26:37.347522 5101 catalog_manager.cc:4841] TS 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591): tablet 2583c878e1104935b41161c87eff6e02 (table test-table [id=e512c8ce70584fdfb8fa76d9ec6bc270]) successfully deleted
W20260929 10:26:37.348695 5102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:43079 (error 108)
I20260929 10:26:37.349334 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 5307
I20260929 10:26:37.353681 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:37.349471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:43079 (local address 127.0.0.1:45768)
0929 10:26:37.353442 (+ 3971us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:37.353535 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3863,"thread_start_us":67,"threads_started":1}
W20260929 10:26:37.353833 5102 catalog_manager.cc:4584] TS b5f4dc09ed9a42d2a00aa6265f9e807f (127.21.33.129:43079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2583c878e1104935b41161c87eff6e02: Network error: Client connection negotiation failed: client connection to 127.21.33.129:43079: connect: Connection refused (error 111)
I20260929 10:26:37.356025 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 5444
I20260929 10:26:37.361101 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 5082
I20260929 10:26:57.366766 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46279
--webserver_interface=127.21.33.190
--webserver_port=46719
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:46279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:26:57.478636 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:57.478893 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:57.478933 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:57.482982 5616 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 10:26:57.483028 5616 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 10:26:57.483044 5616 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 10:26:57.483058 5616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:26:57.483076 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:57.483135 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:57.483157 5616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:26:57.483191 5616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:26:57.488330 5616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:46279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=46719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:57.489406 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:57.490399 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:57.499241 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:57.499261 5616 server_base.cc:1034] running on GCE node
W20260929 10:26:57.499293 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:57.499289 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:57.500097 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:57.501253 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:57.502430 5616 hybrid_clock.cc:648] HybridClock initialized: now 1790677617502398 us; error 48 us; skew 500 ppm
Sep 29 10:26:57 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:57.505126 5616 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:26:57.506035 5616 webserver.cc:460] Webserver started at http://127.21.33.190:46719/ using document root <none> and password file <none>
I20260929 10:26:57.506588 5616 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:57.506664 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:57.510391 5616 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:26:57.512456 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:57.513824 5616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:26:57.513978 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "3e98fefa941c4c2a9aadb17d3c04b396"
format_stamp: "Formatted at 2026-09-29 10:26:14 on dist-test-slave-2rbl"
server_key: "ef85f814435eec64856cc1d2eef67227"
server_key_iv: "5ae54d52dfa2f0e14552f825e5cc4685"
server_key_version: "encryptionkey@0"
I20260929 10:26:57.514518 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:57.526494 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:57.529505 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:57.529714 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:57.536440 5616 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:46279
I20260929 10:26:57.536732 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:46279 every 8 connection(s)
I20260929 10:26:57.537591 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:26:57.541733 5684 sys_catalog.cc:263] Verifying existing consensus state
I20260929 10:26:57.543284 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Bootstrap starting.
I20260929 10:26:57.543469 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5616
I20260929 10:26:57.543959 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:43079
--local_ip_for_outbound_sockets=127.21.33.129
--tserver_master_addrs=127.21.33.190:46279
--webserver_port=34021
--webserver_interface=127.21.33.129
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:26:57.561035 5684 log.cc:824] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:57.572059 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:26:57.572464 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Bootstrap complete.
I20260929 10:26:57.576413 5684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:57.576798 5684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e98fefa941c4c2a9aadb17d3c04b396, State: Initialized, Role: FOLLOWER
I20260929 10:26:57.577342 5684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:57.577461 5684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:26:57.577545 5684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:26:57.577657 5684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 2 FOLLOWER]: Advancing to term 3
I20260929 10:26:57.579473 5684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:57.579808 5684 leader_election.cc:302] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e98fefa941c4c2a9aadb17d3c04b396; no voters:
I20260929 10:26:57.580123 5684 leader_election.cc:288] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 10:26:57.580224 5688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 10:26:57.580420 5688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [term 3 LEADER]: Becoming Leader. State: Replica: 3e98fefa941c4c2a9aadb17d3c04b396, State: Running, Role: LEADER
I20260929 10:26:57.580763 5688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } }
I20260929 10:26:57.581379 5684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:26:57.582140 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } } }
I20260929 10:26:57.582276 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: This master's current role is: LEADER
I20260929 10:26:57.582887 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e98fefa941c4c2a9aadb17d3c04b396. Latest consensus state: current_term: 3 leader_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e98fefa941c4c2a9aadb17d3c04b396" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 46279 } } }
I20260929 10:26:57.583048 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396 [sys.catalog]: This master's current role is: LEADER
I20260929 10:26:57.583333 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:26:57.588984 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:26:57.589735 5697 catalog_manager.cc:1259] Loaded cluster ID: 7b6290d5cf204bbf9878b4d23a061820
I20260929 10:26:57.589802 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:26:57.592756 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:26:57.593560 5697 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Loaded TSK: 0
I20260929 10:26:57.593652 5697 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e98fefa941c4c2a9aadb17d3c04b396: Loaded TSK: 1
F20260929 10:26:57.594365 5697 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f497fe1f9ad google::LogMessage::Fail() at ??:0
@ 0x7f497fe2288a google::LogMessage::SendToLog() at ??:0
@ 0x7f497fe1f4de google::LogMessage::Flush() at ??:0
@ 0x7f497fe20a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4982ae318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4982adbf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f498a7847bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f498a75e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f498a75f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f498a7a0df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f498a7bf82b std::function<>::operator()() at ??:0
@ 0x7f498a75eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f498a75ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f498a75c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f498a7a0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f49820713d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4982071c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f49820733f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4982062512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f49836a66db start_thread at ??:0
@ 0x7f497e91d61f clone at ??:0
W20260929 10:26:57.701272 5686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:57.701568 5686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:57.701639 5686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:57.705911 5686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:57.705981 5686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:57.706089 5686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:26:57.711759 5686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:43079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=34021
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:57.712924 5686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:57.714068 5686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:57.723654 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:57.723640 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:57.723914 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:57.724098 5686 server_base.cc:1034] running on GCE node
I20260929 10:26:57.724570 5686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:57.725397 5686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:57.726608 5686 hybrid_clock.cc:648] HybridClock initialized: now 1790677617726580 us; error 39 us; skew 500 ppm
Sep 29 10:26:57 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:57.729996 5686 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:26:57.731175 5686 webserver.cc:460] Webserver started at http://127.21.33.129:34021/ using document root <none> and password file <none>
I20260929 10:26:57.731873 5686 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:57.731962 5686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:57.737035 5686 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:26:57.739836 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:57.741484 5686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:26:57.741617 5686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "32a87e189d25d7785d78ecb966e9e919"
server_key_iv: "b9f6299bca94afb9e227ace3c50b51d1"
server_key_version: "encryptionkey@0"
I20260929 10:26:57.742326 5686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:57.760183 5686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:57.764454 5686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:57.764668 5686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:57.765410 5686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:57.766743 5748 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 10:26:57.770843 5686 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 10:26:57.770918 5686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260929 10:26:57.770984 5686 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 10:26:57.773248 5686 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 10:26:57.773307 5686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260929 10:26:57.773659 5748 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap starting.
I20260929 10:26:57.784323 5686 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:43079
I20260929 10:26:57.784678 5855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:43079 every 8 connection(s)
I20260929 10:26:57.785553 5686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:26:57.791657 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5686
I20260929 10:26:57.792171 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:41489
--local_ip_for_outbound_sockets=127.21.33.130
--tserver_master_addrs=127.21.33.190:46279
--webserver_port=41727
--webserver_interface=127.21.33.130
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:26:57.793469 5748 log.cc:824] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:26:57.802076 5748 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 10:26:57.802589 5748 tablet_bootstrap.cc:492] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Bootstrap complete.
I20260929 10:26:57.804128 5748 ts_tablet_manager.cc:1397] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.009s
I20260929 10:26:57.808084 5748 raft_consensus.cc:348] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:57.808655 5748 raft_consensus.cc:729] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f4dc09ed9a42d2a00aa6265f9e807f, State: Initialized, Role: FOLLOWER
I20260929 10:26:57.809234 5748 consensus_queue.cc:260] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:57.810297 5748 ts_tablet_manager.cc:1428] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260929 10:26:57.828348 5858 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:26:57.787969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:55855)
0929 10:26:57.788603 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:57.788663 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:57.789868 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:57.827549 (+ 37681us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":162,"threads_started":1}
W20260929 10:26:57.828866 5856 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46279 (0 consecutive failures): Network error: Failed to ping master at 127.21.33.190:46279: Client connection negotiation failed: client connection to 127.21.33.190:46279: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 10:26:57.829730 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:57.829437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:60491)
0929 10:26:57.829599 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:57.829673 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 10:26:57.830389 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:57.830186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:52589)
0929 10:26:57.830292 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:57.830343 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:26:57.830551 5856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:26:57.913183 5859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:57.913537 5859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:57.913620 5859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:57.917702 5859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:57.917784 5859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:57.917873 5859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:26:57.922993 5859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:41489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=41727
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:57.923992 5859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:57.925011 5859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:26:57.935004 5859 server_base.cc:1034] running on GCE node
W20260929 10:26:57.935011 5869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:57.934957 5866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:57.934962 5867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:57.935582 5859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:57.936208 5859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:57.937388 5859 hybrid_clock.cc:648] HybridClock initialized: now 1790677617937358 us; error 43 us; skew 500 ppm
Sep 29 10:26:57 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:57.940182 5859 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:26:57.941093 5859 webserver.cc:460] Webserver started at http://127.21.33.130:41727/ using document root <none> and password file <none>
I20260929 10:26:57.941689 5859 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:57.941794 5859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:57.945709 5859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:26:57.947921 5876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:57.949411 5859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:26:57.949542 5859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "fc11f8ec6e2846a1866cb832daa7d43e"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "06f9dce6ca37662afd58bcc3b426e02e"
server_key_iv: "5ec5a8feecba8861c78f2bb66309c3bb"
server_key_version: "encryptionkey@0"
I20260929 10:26:57.950114 5859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:57.976334 5859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:57.979398 5859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:57.979612 5859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:57.980221 5859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:57.981042 5859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:26:57.981112 5859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:57.981178 5859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:26:57.981256 5859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:57.989995 5859 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:41489
I20260929 10:26:57.990039 5989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:41489 every 8 connection(s)
I20260929 10:26:57.991040 5859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:26:57.992377 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5859
I20260929 10:26:57.992811 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:39591
--local_ip_for_outbound_sockets=127.21.33.131
--tserver_master_addrs=127.21.33.190:46279
--webserver_port=39753
--webserver_interface=127.21.33.131
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:26:57.995261 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:57.993352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:58957)
0929 10:26:57.993853 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:57.994000 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":104,"threads_started":1}
W20260929 10:26:57.995715 5990 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46279 (0 consecutive failures): Network error: Failed to ping master at 127.21.33.190:46279: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
I20260929 10:26:57.996305 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:57.996056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:38887)
0929 10:26:57.996210 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:57.996253 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 10:26:57.996889 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:57.996693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:43837)
0929 10:26:57.996794 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:57.996842 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 10:26:57.997040 5990 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 10:26:58.102101 5993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:26:58.102370 5993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:26:58.102440 5993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:26:58.106441 5993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:26:58.106525 5993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:26:58.106616 5993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:26:58.111616 5993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:39591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=39753
--tserver_master_addrs=127.21.33.190:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:26:58.112622 5993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:26:58.113683 5993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:26:58.123128 6001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:58.123306 5993 server_base.cc:1034] running on GCE node
W20260929 10:26:58.123064 5998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:26:58.123064 5999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:26:58.123837 5993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:26:58.124488 5993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:26:58.125672 5993 hybrid_clock.cc:648] HybridClock initialized: now 1790677618125646 us; error 48 us; skew 500 ppm
Sep 29 10:26:58 dist-test-slave-2rbl krb5kdc[4535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677618, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:26:58.128407 5993 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:26:58.129371 5993 webserver.cc:460] Webserver started at http://127.21.33.131:39753/ using document root <none> and password file <none>
I20260929 10:26:58.129923 5993 fs_manager.cc:359] Metadata directory not provided
I20260929 10:26:58.130013 5993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:26:58.133899 5993 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 10:26:58.136119 6008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:58.137482 5993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:26:58.137611 5993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "83e4820d9d4c4989a22c29ebbb3b85dc"
format_stamp: "Formatted at 2026-09-29 10:26:15 on dist-test-slave-2rbl"
server_key: "f78c9baa69f0e4b1cadbd42bc814d355"
server_key_iv: "6fe890d726c8c7ff45eafab9aa856521"
server_key_version: "encryptionkey@0"
I20260929 10:26:58.138164 5993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:26:58.165634 5993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:26:58.168898 5993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:26:58.169116 5993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:26:58.169767 5993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:26:58.170615 5993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:26:58.170689 5993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:58.170759 5993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:26:58.170810 5993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:26:58.179988 5993 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:39591
I20260929 10:26:58.180022 6121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:39591 every 8 connection(s)
I20260929 10:26:58.181003 5993 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:26:58.184803 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:58.183091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:53639)
0929 10:26:58.183540 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:58.183658 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
W20260929 10:26:58.185289 6122 heartbeater.cc:644] Failed to heartbeat to 127.21.33.190:46279 (0 consecutive failures): Network error: Failed to ping master at 127.21.33.190:46279: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
I20260929 10:26:58.185882 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:58.185661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:59313)
0929 10:26:58.185769 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:58.185819 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 10:26:58.186508 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:58.186305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:46811)
0929 10:26:58.186410 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:58.186453 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 10:26:58.186816 6122 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 10:26:58.189810 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 5993
I20260929 10:26:58.831684 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:58.831230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:36661)
0929 10:26:58.831518 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:58.831577 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260929 10:26:58.998283 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:58.997796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:43279)
0929 10:26:58.998101 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:58.998169 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260929 10:26:59.187786 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:59.187405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:39825)
0929 10:26:59.187632 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:59.187691 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260929 10:26:59.538056 6129 raft_consensus.cc:482] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:26:59.538273 6129 raft_consensus.cc:504] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:26:59.539431 6129 leader_election.cc:288] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591), fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
Sep 29 10:26:59 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.131@KRBTEST.COM
Sep 29 10:26:59 dist-test-slave-2rbl krb5kdc[4535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677617, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.130@KRBTEST.COM
I20260929 10:26:59.556185 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:59.539869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:56321 (local address 127.21.33.131:39591)
0929 10:26:59.540329 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:59.540349 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:59.540385 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:26:59.541149 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:59.541156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:59.541570 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:59.541916 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:59.543493 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.545027 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.547063 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.547795 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.548024 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:59.551250 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:59.551258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:59.551276 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:59.551318 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:59.553388 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:59.554080 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:59.554090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:59.554094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:59.554212 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:59.554651 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:59.554653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:59.554655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:59.555039 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:59.555185 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:59.555728 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 10:26:59.555986 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":118,"threads_started":1}
I20260929 10:26:59.556185 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:59.539749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:41489 (local address 127.21.33.129:40187)
0929 10:26:59.540097 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:59.540113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:59.540201 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:59.541932 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:59.541939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:59.542530 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:59.543293 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:59.543315 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.545177 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.545181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:59.546872 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:59.546884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.547967 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.547971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:59.548258 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:59.549064 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:59.549077 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:59.551251 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:59.553598 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:59.553600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:59.553602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:59.553951 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:59.554332 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:59.554338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:59.554340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:59.554518 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:59.555164 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:59.555171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:59.555418 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 10:26:59.555614 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 10:26:59.555996 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":334,"spinlock_wait_cycles":27264,"thread_start_us":73,"threads_started":1}
I20260929 10:26:59.556216 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:59.540000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:40187 (local address 127.21.33.130:41489)
0929 10:26:59.540349 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 10:26:59.540360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:26:59.540394 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:26:59.541163 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:26:59.541175 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:26:59.541570 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:26:59.541876 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:26:59.543493 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.545015 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.547032 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.547795 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.548024 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:59.551437 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:26:59.551442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:26:59.551455 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:26:59.551484 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:26:59.553480 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:59.554069 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:59.554076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:59.554084 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:59.554212 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:26:59.554623 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:26:59.554629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:26:59.554632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:26:59.555040 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:26:59.555185 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 10:26:59.555727 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 10:26:59.556010 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":72,"threads_started":1}
I20260929 10:26:59.556324 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:59.539577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:39591 (local address 127.21.33.129:56321)
0929 10:26:59.539884 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:59.539899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:26:59.540011 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:26:59.541930 (+ 1919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:26:59.541952 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:26:59.542530 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:26:59.543293 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:59.543315 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.545177 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.545181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:59.546873 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:26:59.546884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:26:59.547967 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:26:59.547971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:26:59.548256 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:26:59.549047 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:26:59.549078 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:26:59.550986 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:26:59.553507 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:59.553516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:59.553532 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:59.553952 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:59.554315 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:26:59.554318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:26:59.554339 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:26:59.554524 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:26:59.555164 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:26:59.555171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:26:59.555422 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 10:26:59.555614 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 10:26:59.556128 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":591,"spinlock_wait_cycles":47744,"thread_start_us":75,"threads_started":1}
W20260929 10:26:59.558868 5743 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
W20260929 10:26:59.559439 5744 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
I20260929 10:26:59.559514 5744 leader_election.cc:302] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5f4dc09ed9a42d2a00aa6265f9e807f; no voters: 83e4820d9d4c4989a22c29ebbb3b85dc, fc11f8ec6e2846a1866cb832daa7d43e
I20260929 10:26:59.559737 6129 raft_consensus.cc:2726] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:26:59.832790 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:59.832513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:46883)
0929 10:26:59.832648 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:59.832706 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 10:26:59.999449 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:26:59.999031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:53331)
0929 10:26:59.999287 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 10:26:59.999340 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260929 10:27:00.188995 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:00.188574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:53843)
0929 10:27:00.188840 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:00.188897 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260929 10:27:00.833998 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:00.833579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:50621)
0929 10:27:00.833841 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:00.833905 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":110,"threads_started":1}
I20260929 10:27:01.000627 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:01.000168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:43845)
0929 10:27:01.000463 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:01.000528 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260929 10:27:01.190169 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:01.189749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:32911)
0929 10:27:01.190000 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:01.190050 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260929 10:27:01.586930 6139 raft_consensus.cc:482] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:01.587141 6139 raft_consensus.cc:504] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:27:01.587575 6139 leader_election.cc:288] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591), fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
W20260929 10:27:01.588490 5743 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
W20260929 10:27:01.588690 5744 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
I20260929 10:27:01.588788 5744 leader_election.cc:302] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5f4dc09ed9a42d2a00aa6265f9e807f; no voters: 83e4820d9d4c4989a22c29ebbb3b85dc, fc11f8ec6e2846a1866cb832daa7d43e
I20260929 10:27:01.588989 6139 raft_consensus.cc:2726] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:27:01.835171 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:01.834784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:35043)
0929 10:27:01.835018 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:01.835077 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260929 10:27:02.001946 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:02.001488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:42819)
0929 10:27:02.001785 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:02.001846 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260929 10:27:02.191340 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:02.190911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:38113)
0929 10:27:02.191190 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:02.191251 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260929 10:27:02.836853 6143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:02.836268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:57005)
0929 10:27:02.836633 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:02.836725 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":90,"threads_started":1}
I20260929 10:27:03.003365 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:03.002946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:36669)
0929 10:27:03.003220 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:03.003271 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260929 10:27:03.192548 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:03.192146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:44189)
0929 10:27:03.192392 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:03.192448 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260929 10:27:03.838120 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:03.837665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:55233)
0929 10:27:03.837958 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:03.838019 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260929 10:27:04.004804 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:04.004290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:43833)
0929 10:27:04.004603 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:04.004699 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
I20260929 10:27:04.193843 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:04.193403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:59821)
0929 10:27:04.193651 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:04.193711 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260929 10:27:04.348191 6149 raft_consensus.cc:482] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:04.348349 6149 raft_consensus.cc:504] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:27:04.348805 6149 leader_election.cc:288] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591), fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
W20260929 10:27:04.349604 5743 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
W20260929 10:27:04.349745 5744 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
I20260929 10:27:04.349845 5744 leader_election.cc:302] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5f4dc09ed9a42d2a00aa6265f9e807f; no voters: 83e4820d9d4c4989a22c29ebbb3b85dc, fc11f8ec6e2846a1866cb832daa7d43e
I20260929 10:27:04.349977 6149 raft_consensus.cc:2726] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:27:04.839277 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:04.838862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:53557)
0929 10:27:04.839133 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:04.839184 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260929 10:27:05.006062 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:05.005630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:39019)
0929 10:27:05.005913 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:05.005970 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":66,"threads_started":1}
I20260929 10:27:05.195094 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:05.194634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:35929)
0929 10:27:05.194948 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:05.195003 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
I20260929 10:27:05.840431 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:05.840037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:42455)
0929 10:27:05.840280 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:05.840335 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260929 10:27:06.007261 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:06.006849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:51969)
0929 10:27:06.007099 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:06.007156 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260929 10:27:06.196432 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:06.195942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:60343)
0929 10:27:06.196267 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:06.196333 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":116,"threads_started":1}
I20260929 10:27:06.841876 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:06.841312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:36549)
0929 10:27:06.841667 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:06.841747 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":106,"threads_started":1}
I20260929 10:27:07.008378 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:07.008005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:37425)
0929 10:27:07.008225 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:07.008285 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260929 10:27:07.197710 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:07.197278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:54615)
0929 10:27:07.197533 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:07.197605 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260929 10:27:07.843333 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:07.842872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:56887)
0929 10:27:07.843163 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:07.843235 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260929 10:27:08.009627 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:08.009166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:43659)
0929 10:27:08.009470 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:08.009521 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260929 10:27:08.199230 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:08.198582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:54379)
0929 10:27:08.198955 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:08.199060 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":76,"threads_started":1}
I20260929 10:27:08.320644 6162 raft_consensus.cc:482] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:08.320781 6162 raft_consensus.cc:504] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:27:08.321249 6162 leader_election.cc:288] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591), fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
W20260929 10:27:08.322019 5744 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
W20260929 10:27:08.322151 5743 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
I20260929 10:27:08.322211 5744 leader_election.cc:302] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5f4dc09ed9a42d2a00aa6265f9e807f; no voters: 83e4820d9d4c4989a22c29ebbb3b85dc, fc11f8ec6e2846a1866cb832daa7d43e
I20260929 10:27:08.322347 6162 raft_consensus.cc:2726] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:27:08.844504 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:08.844100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:51683)
0929 10:27:08.844354 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:08.844410 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260929 10:27:09.010775 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:09.010376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:33293)
0929 10:27:09.010621 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:09.010676 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260929 10:27:09.200480 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:09.200032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:42489)
0929 10:27:09.200319 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:09.200382 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
I20260929 10:27:09.845839 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:09.845431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:57415)
0929 10:27:09.845689 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:09.845743 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260929 10:27:10.011932 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:10.011528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:53449)
0929 10:27:10.011780 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:10.011836 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260929 10:27:10.080777 6168 raft_consensus.cc:482] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:10.080909 6168 raft_consensus.cc:504] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e4820d9d4c4989a22c29ebbb3b85dc" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39591 } } peers { permanent_uuid: "fc11f8ec6e2846a1866cb832daa7d43e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 41489 } } peers { permanent_uuid: "b5f4dc09ed9a42d2a00aa6265f9e807f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 43079 } }
I20260929 10:27:10.081347 6168 leader_election.cc:288] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591), fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489)
W20260929 10:27:10.082173 5744 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83e4820d9d4c4989a22c29ebbb3b85dc (127.21.33.131:39591): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
W20260929 10:27:10.082263 5743 leader_election.cc:341] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc11f8ec6e2846a1866cb832daa7d43e (127.21.33.130:41489): Not found: Tablet not found: 2583c878e1104935b41161c87eff6e02
I20260929 10:27:10.082324 5743 leader_election.cc:302] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5f4dc09ed9a42d2a00aa6265f9e807f; no voters: 83e4820d9d4c4989a22c29ebbb3b85dc, fc11f8ec6e2846a1866cb832daa7d43e
I20260929 10:27:10.082508 6168 raft_consensus.cc:2726] T 2583c878e1104935b41161c87eff6e02 P b5f4dc09ed9a42d2a00aa6265f9e807f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 10:27:10.201865 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:10.201395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:60409)
0929 10:27:10.201707 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:10.201770 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
I20260929 10:27:10.847008 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:10.846577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:47513)
0929 10:27:10.846855 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:10.846914 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260929 10:27:11.013332 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:11.012830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:41269)
0929 10:27:11.013106 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:11.013174 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260929 10:27:11.202970 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:11.202569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:53975)
0929 10:27:11.202827 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:11.202878 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260929 10:27:11.848255 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:11.847763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:47821)
0929 10:27:11.848036 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:11.848140 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260929 10:27:12.014436 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:12.014037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:58521)
0929 10:27:12.014287 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:12.014342 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260929 10:27:12.204197 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:12.203743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:55771)
0929 10:27:12.204033 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:12.204093 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260929 10:27:12.849749 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:12.849278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.129:51869)
0929 10:27:12.849598 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:12.849657 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":134,"threads_started":1}
I20260929 10:27:13.015865 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:13.015370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.130:60457)
0929 10:27:13.015706 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:13.015767 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260929 10:27:13.199122 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 5686
I20260929 10:27:13.205030 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 5859
I20260929 10:27:13.206182 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:13.205269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:46279 (local address 127.21.33.131:36801)
0929 10:27:13.205975 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:13.206047 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.190:46279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":70,"threads_started":1}
I20260929 10:27:13.210894 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 5993
2026-09-29T10:27:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59630 ms)
[----------] 30 tests from SecurityITest (245369 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:13 dist-test-slave-2rbl krb5kdc[6181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:13 dist-test-slave-2rbl krb5kdc[6181](info): set up 2 sockets
Sep 29 10:27:13 dist-test-slave-2rbl krb5kdc[6181](info): commencing operation
krb5kdc: starting...
W20260929 10:27:14.249706 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:14 dist-test-slave-2rbl krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:14.376451 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39781
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:14.488364 6194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:14.488626 6194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:14.488667 6194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:14.492563 6194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:14.492626 6194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:14.492645 6194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:14.492668 6194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:14.492704 6194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:14.498018 6194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:14.499037 6194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:14.500013 6194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:14.508495 6200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:14.508569 6194 server_base.cc:1034] running on GCE node
W20260929 10:27:14.508535 6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:14.508687 6199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:14.509407 6194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:14.510533 6194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:14.511713 6194 hybrid_clock.cc:648] HybridClock initialized: now 1790677634511702 us; error 25 us; skew 500 ppm
Sep 29 10:27:14 dist-test-slave-2rbl krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:14.514562 6194 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:14.515448 6194 webserver.cc:460] Webserver started at http://127.21.33.190:38955/ using document root <none> and password file <none>
I20260929 10:27:14.515969 6194 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:14.516027 6194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:14.516186 6194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:14.518281 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba"
format_stamp: "Formatted at 2026-09-29 10:27:14 on dist-test-slave-2rbl"
server_key: "d6fda1e8d7332842620bf63f3b9727c4"
server_key_iv: "949ab838a826892dd18dc22692beed15"
server_key_version: "encryptionkey@0"
I20260929 10:27:14.518814 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba"
format_stamp: "Formatted at 2026-09-29 10:27:14 on dist-test-slave-2rbl"
server_key: "d6fda1e8d7332842620bf63f3b9727c4"
server_key_iv: "949ab838a826892dd18dc22692beed15"
server_key_version: "encryptionkey@0"
I20260929 10:27:14.522421 6194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:27:14.524825 6209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:14.526165 6194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:14.526292 6194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba"
format_stamp: "Formatted at 2026-09-29 10:27:14 on dist-test-slave-2rbl"
server_key: "d6fda1e8d7332842620bf63f3b9727c4"
server_key_iv: "949ab838a826892dd18dc22692beed15"
server_key_version: "encryptionkey@0"
I20260929 10:27:14.526402 6194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:14.563019 6194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:14.566108 6194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:14.566316 6194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:14.573642 6194 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:39781
I20260929 10:27:14.573710 6261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:39781 every 8 connection(s)
I20260929 10:27:14.574632 6194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:14.577431 6262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:14.582857 6262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba: Bootstrap starting.
I20260929 10:27:14.583727 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 6194
I20260929 10:27:14.583868 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:14.584187 21638 external_mini_cluster.cc:1428] Setting key fcd78bc2fd1902684821dc1511bd0dee
I20260929 10:27:14.585907 6262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:14.586650 6262 log.cc:824] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:14.588840 6262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba: No bootstrap required, opened a new log
Sep 29 10:27:14 dist-test-slave-2rbl krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:14.593573 6262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39781 } }
I20260929 10:27:14.593827 6262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:14.593887 6262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a937aeef8fd4e7fb4708a3b5cfdbdba, State: Initialized, Role: FOLLOWER
I20260929 10:27:14.594338 6262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39781 } }
I20260929 10:27:14.594488 6262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:14.594583 6262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:14.594691 6262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:14.595914 6262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39781 } }
I20260929 10:27:14.596302 6262 leader_election.cc:302] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a937aeef8fd4e7fb4708a3b5cfdbdba; no voters:
I20260929 10:27:14.596674 6262 leader_election.cc:288] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:14.596787 6267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:14.597034 6267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [term 1 LEADER]: Becoming Leader. State: Replica: 5a937aeef8fd4e7fb4708a3b5cfdbdba, State: Running, Role: LEADER
I20260929 10:27:14.597445 6267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39781 } }
I20260929 10:27:14.598085 6262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:14.599052 6268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39781 } } }
I20260929 10:27:14.599175 6268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:14.599509 6269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a937aeef8fd4e7fb4708a3b5cfdbdba. Latest consensus state: current_term: 1 leader_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a937aeef8fd4e7fb4708a3b5cfdbdba" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39781 } } }
I20260929 10:27:14.599581 6269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:14.599999 6276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:14.600005 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:14.585417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.21.33.190:39781)
0929 10:27:14.585835 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:14.585850 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:14.585885 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:27:14.586562 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:14.586580 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:14.586974 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:14.587366 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:14.588207 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:14.589290 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:14.590420 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:14.590888 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:14.591004 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:14.593917 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:14.593924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:14.593942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:14.593980 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:14.596067 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:14.596514 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:14.596520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:14.596527 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:14.596689 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:14.597031 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:14.597035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:14.597036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:14.597546 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:14.597769 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:14.598502 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 10:27:14.598836 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260929 10:27:14.603407 6276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:14.609022 6276 catalog_manager.cc:1347] Generated new cluster ID: e4cefd98404440ae8e84f33f22c1ed7d
I20260929 10:27:14.609117 6276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:14.625157 6276 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:14.626561 6276 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:14.633781 6276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba: Generated new TSK 0
I20260929 10:27:14.634591 6276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:27:14.672354 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39781
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:14.785755 6288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:14.786072 6288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:14.786160 6288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:14.790227 6288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:14.790329 6288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:14.790436 6288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:27:14.795914 6288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:14.797055 6288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:14.798228 6288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:14.807938 6294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:14.807931 6296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:14.807945 6293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:14.807984 6288 server_base.cc:1034] running on GCE node
I20260929 10:27:14.808598 6288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:14.809296 6288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:14.810487 6288 hybrid_clock.cc:648] HybridClock initialized: now 1790677634810471 us; error 39 us; skew 500 ppm
Sep 29 10:27:14 dist-test-slave-2rbl krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:14.813342 6288 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:27:14.814182 6288 webserver.cc:460] Webserver started at http://127.21.33.129:36437/ using document root <none> and password file <none>
I20260929 10:27:14.814762 6288 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:14.814854 6288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:14.815060 6288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:14.817080 6288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "b73a5a5c10a74c5597c4c2f368b547dc"
format_stamp: "Formatted at 2026-09-29 10:27:14 on dist-test-slave-2rbl"
server_key: "b5749600d6e70401a20b77ace17a6e86"
server_key_iv: "6fd3fa42ed285125d119d715b101f6f4"
server_key_version: "encryptionkey@0"
I20260929 10:27:14.817746 6288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "b73a5a5c10a74c5597c4c2f368b547dc"
format_stamp: "Formatted at 2026-09-29 10:27:14 on dist-test-slave-2rbl"
server_key: "b5749600d6e70401a20b77ace17a6e86"
server_key_iv: "6fd3fa42ed285125d119d715b101f6f4"
server_key_version: "encryptionkey@0"
I20260929 10:27:14.821591 6288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:27:14.823868 6303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:14.825078 6288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:14.825251 6288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "b73a5a5c10a74c5597c4c2f368b547dc"
format_stamp: "Formatted at 2026-09-29 10:27:14 on dist-test-slave-2rbl"
server_key: "b5749600d6e70401a20b77ace17a6e86"
server_key_iv: "6fd3fa42ed285125d119d715b101f6f4"
server_key_version: "encryptionkey@0"
I20260929 10:27:14.825371 6288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:14.843263 6288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:14.846233 6288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:14.846438 6288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:14.847048 6288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:14.847870 6288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:14.847940 6288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:14.848011 6288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:14.848064 6288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:14.857075 6288 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:44225
I20260929 10:27:14.857098 6416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:44225 every 8 connection(s)
I20260929 10:27:14.858047 6288 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:27:14.859073 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 6288
I20260929 10:27:14.859215 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:27:14.859532 21638 external_mini_cluster.cc:1428] Setting key 9f5ebc2afccd2e2b88215d86cb5044ac
Sep 29 10:27:14 dist-test-slave-2rbl krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:27:14.873044 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:14.860122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:32995 (local address 127.21.33.190:39781)
0929 10:27:14.860277 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:14.860283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:14.861329 (+ 1046us) server_negotiation.cc:407] Connection header received
0929 10:27:14.862833 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:14.862836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:14.862891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:14.862971 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:14.864139 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:14.864685 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:14.866031 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:14.866481 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:14.866580 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:14.869321 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:14.869325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:14.869328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:14.869362 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:14.870825 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:14.871412 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:14.871416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:14.871417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:14.871503 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:14.871910 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:14.871913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:14.871915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:14.872111 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:14.872220 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:14.872741 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 10:27:14.872873 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:27:14.873978 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:14.860604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39781 (local address 127.21.33.129:32995)
0929 10:27:14.861148 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:14.861229 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:14.862615 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:14.863154 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:14.863161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:14.863512 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:14.863989 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:14.864002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:14.864817 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:14.864820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:14.865873 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:14.865885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:14.866816 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:14.866820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:14.867002 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:14.867638 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:14.867656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:14.869108 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:14.870969 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:14.870976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:14.870994 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:14.871297 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:14.871605 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:14.871609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:14.871611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:14.871808 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:14.872219 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:14.872227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:14.872499 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 10:27:14.872687 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 10:27:14.873038 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":176,"threads_started":1}
I20260929 10:27:14.875319 6417 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39781
I20260929 10:27:14.875617 6417 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:14.876281 6417 heartbeater.cc:511] Master 127.21.33.190:39781 requested a full tablet report, sending...
I20260929 10:27:14.878052 6226 ts_manager.cc:194] Registered new tserver with Master: b73a5a5c10a74c5597c4c2f368b547dc (127.21.33.129:44225)
I20260929 10:27:14.879216 6226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:32995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:27:14.888365 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39781
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:14.999979 6422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:15.000351 6422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:15.000432 6422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:15.004642 6422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:15.004735 6422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:15.004837 6422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:27:15.010221 6422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:15.011276 6422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:15.012326 6422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:15.022135 6428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:15.022090 6430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:15.022333 6422 server_base.cc:1034] running on GCE node
W20260929 10:27:15.022091 6427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:15.022948 6422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:15.023608 6422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:15.024807 6422 hybrid_clock.cc:648] HybridClock initialized: now 1790677635024791 us; error 34 us; skew 500 ppm
Sep 29 10:27:15 dist-test-slave-2rbl krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:15.027529 6422 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:27:15.028489 6422 webserver.cc:460] Webserver started at http://127.21.33.130:46179/ using document root <none> and password file <none>
I20260929 10:27:15.029048 6422 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:15.029141 6422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:15.029426 6422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:15.031659 6422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "d40a928cd53240fb9ad160a89840ed91"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "4db452a7fb9f903b4ad762f66722ac10"
server_key_iv: "40b4cf9774fb4f51872ec2c8549710b2"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.032215 6422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "d40a928cd53240fb9ad160a89840ed91"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "4db452a7fb9f903b4ad762f66722ac10"
server_key_iv: "40b4cf9774fb4f51872ec2c8549710b2"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.035974 6422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:27:15.038354 6437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:15.039565 6422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:27:15.039714 6422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "d40a928cd53240fb9ad160a89840ed91"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "4db452a7fb9f903b4ad762f66722ac10"
server_key_iv: "40b4cf9774fb4f51872ec2c8549710b2"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.040016 6422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:15.055150 6422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:15.058224 6422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:15.058431 6422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:15.059020 6422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:15.059856 6422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:15.059930 6422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:15.059998 6422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:15.060050 6422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:15.068773 6422 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:33467
I20260929 10:27:15.068832 6550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:33467 every 8 connection(s)
I20260929 10:27:15.069773 6422 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:27:15.075006 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 6422
I20260929 10:27:15.075203 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:27:15.075474 21638 external_mini_cluster.cc:1428] Setting key 679e788dd1b5ba1160fd48dc4d08863a
Sep 29 10:27:15 dist-test-slave-2rbl krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
I20260929 10:27:15.084578 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.071649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:58013 (local address 127.21.33.190:39781)
0929 10:27:15.071795 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.071798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.072707 (+ 909us) server_negotiation.cc:407] Connection header received
0929 10:27:15.074226 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.074230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.074293 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.074407 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:15.075732 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.076397 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.077706 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.077937 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.078057 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.080809 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:15.080812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:15.080814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:15.080839 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:15.082436 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:15.082988 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:15.082996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:15.082998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:15.083072 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:15.083425 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:15.083430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:15.083432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:15.083661 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:15.083782 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.084283 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.084412 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:27:15.085497 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.072020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39781 (local address 127.21.33.130:58013)
0929 10:27:15.072527 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.072579 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:15.073939 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:15.074585 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:15.074592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:15.074955 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:15.075567 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.075589 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.076549 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.076554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.077566 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.077576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.078113 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.078118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.078293 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.079004 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:15.079029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:15.080613 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:15.082579 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:15.082585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:15.082600 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:15.082867 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:15.083179 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:15.083183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:15.083185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:15.083327 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:15.083778 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:15.083784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:15.084043 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 10:27:15.084240 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 10:27:15.084567 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260929 10:27:15.086688 6551 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
I20260929 10:27:15.086947 6551 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:15.087446 6551 heartbeater.cc:511] Master 127.21.33.190:39781 requested a full tablet report, sending...
I20260929 10:27:15.088579 6226 ts_manager.cc:194] Registered new tserver with Master: d40a928cd53240fb9ad160a89840ed91 (127.21.33.130:33467)
I20260929 10:27:15.089439 6226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:58013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:27:15.103777 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39781
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:15.220242 6556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:15.220479 6556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:15.220520 6556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:15.224578 6556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:15.224637 6556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:15.224736 6556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:27:15.230486 6556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:15.231577 6556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:15.232697 6556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:15.242851 6564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:15.242908 6556 server_base.cc:1034] running on GCE node
W20260929 10:27:15.242888 6561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:15.242862 6562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:15.243569 6556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:15.244272 6556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:15.245451 6556 hybrid_clock.cc:648] HybridClock initialized: now 1790677635245439 us; error 25 us; skew 500 ppm
Sep 29 10:27:15 dist-test-slave-2rbl krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:15.248117 6556 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:27:15.248912 6556 webserver.cc:460] Webserver started at http://127.21.33.131:45175/ using document root <none> and password file <none>
I20260929 10:27:15.249547 6556 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:15.249609 6556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:15.249781 6556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:15.251705 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "f6133a81473941d4901b3f7fbaca9537"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "198edf13bd19fdd1d0ce5dfe14713b9c"
server_key_iv: "26ab417c79ca557c6b92e87fe94ef7de"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.252215 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "f6133a81473941d4901b3f7fbaca9537"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "198edf13bd19fdd1d0ce5dfe14713b9c"
server_key_iv: "26ab417c79ca557c6b92e87fe94ef7de"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.255822 6556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:27:15.258301 6571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:15.260202 6556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:27:15.260329 6556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "f6133a81473941d4901b3f7fbaca9537"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "198edf13bd19fdd1d0ce5dfe14713b9c"
server_key_iv: "26ab417c79ca557c6b92e87fe94ef7de"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.260449 6556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:15.282989 6556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:15.285980 6556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:15.286185 6556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:15.286804 6556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:15.287629 6556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:15.287675 6556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:15.287755 6556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:15.287796 6556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:15.296881 6556 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:45351
I20260929 10:27:15.296905 6684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:45351 every 8 connection(s)
I20260929 10:27:15.297842 6556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:27:15.300312 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 6556
I20260929 10:27:15.300441 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:27:15.300781 21638 external_mini_cluster.cc:1428] Setting key 33a4f5399733d7fbfae477d43e5b11b6
Sep 29 10:27:15 dist-test-slave-2rbl krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:15.311383 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.299854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:50353 (local address 127.21.33.190:39781)
0929 10:27:15.300021 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.300025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.301066 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 10:27:15.302661 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.302663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.302703 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.302762 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:15.303869 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.304393 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.305588 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.305881 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.305969 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.308295 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:15.308298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:15.308300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:15.308323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:15.309671 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:15.310149 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:15.310152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:15.310153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:15.310226 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:15.310501 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:15.310503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:15.310504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:15.310672 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:15.310744 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.311177 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.311265 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 10:27:15.312170 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.300365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39781 (local address 127.21.33.131:50353)
0929 10:27:15.300903 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.300960 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:15.302458 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:15.302891 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:15.302897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:15.303238 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:15.303742 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.303755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.304492 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.304494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.305444 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.305453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.306009 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.306012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.306146 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.306681 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:15.306696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:15.308142 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:15.309788 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:15.309792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:15.309803 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:15.310046 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:15.310302 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:15.310304 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:15.310305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:15.310426 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:15.310753 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:15.310758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:15.310968 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 10:27:15.311114 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 10:27:15.311347 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260929 10:27:15.313431 6685 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39781
I20260929 10:27:15.313659 6685 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:15.314258 6685 heartbeater.cc:511] Master 127.21.33.190:39781 requested a full tablet report, sending...
I20260929 10:27:15.315299 6226 ts_manager.cc:194] Registered new tserver with Master: f6133a81473941d4901b3f7fbaca9537 (127.21.33.131:45351)
I20260929 10:27:15.315871 6226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:50353
I20260929 10:27:15.326308 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:27:15.335625 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.328193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.21.33.190:39781)
0929 10:27:15.328351 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.328355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.328523 (+ 168us) server_negotiation.cc:407] Connection header received
0929 10:27:15.328617 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.328619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.328659 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.328722 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:15.329591 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.330106 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.330946 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.331167 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.331267 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.332503 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:15.332506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:15.332508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:15.332531 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:15.333961 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:15.334369 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:15.334373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:15.334375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:15.334457 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:15.334738 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:15.334742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:15.334744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:15.334957 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:15.335096 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.335338 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.335453 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 10:27:15.338404 6226 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:27:15.341379 6226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:27:15.355161 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.349266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44225 (local address 127.0.0.1:52440)
0929 10:27:15.350039 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.350092 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:15.350222 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:15.350847 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:15.350853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:15.350879 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:15.351009 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.351015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.352198 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.352203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.353424 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.353434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.354587 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.354591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.354706 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.354748 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 10:27:15.354867 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 10:27:15.354997 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":46,"thread_start_us":77,"threads_started":1}
I20260929 10:27:15.355192 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.349399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.21.33.129:44225)
0929 10:27:15.349697 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.349716 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.350363 (+ 647us) server_negotiation.cc:407] Connection header received
0929 10:27:15.350454 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.350461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.350700 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.350854 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:15.351125 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.352071 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.353558 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.354460 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.354571 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.354715 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.354921 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.355025 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":135,"threads_started":1}
I20260929 10:27:15.355608 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.349960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33467 (local address 127.0.0.1:55552)
0929 10:27:15.350520 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.350532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:15.350605 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:15.351298 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:15.351301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:15.351318 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:15.351410 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.351416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.352797 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.352800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.354003 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.354013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.355181 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.355187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.355293 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.355325 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 10:27:15.355401 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 10:27:15.355469 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":43,"thread_start_us":83,"threads_started":1}
I20260929 10:27:15.355829 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.349382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:45351 (local address 127.0.0.1:56332)
0929 10:27:15.350107 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.350121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:15.350222 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:15.351464 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:15.351466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:15.351483 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:15.351569 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.351575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.353088 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.353092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.354124 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.354135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.355426 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.355429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.355530 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.355552 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:27:15.355637 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 10:27:15.355704 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":30,"thread_start_us":77,"threads_started":1}
I20260929 10:27:15.355891 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.350043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.21.33.131:45351)
0929 10:27:15.350847 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.350859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.350879 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:27:15.350950 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.350955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.351164 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.351313 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:15.351697 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.352953 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.354249 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.355077 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.355178 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.355547 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.355670 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.355762 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":144,"threads_started":1}
I20260929 10:27:15.356076 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.350457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.21.33.130:33467)
0929 10:27:15.350851 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.350862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.350884 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:27:15.350956 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.350961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.351173 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.351328 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:15.351502 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.352657 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.354137 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.354888 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.354979 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.355105 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.355856 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.355950 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":94,"threads_started":1}
I20260929 10:27:15.358319 6351 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca2d0474e8146f3a6c51d71fc31842d (DEFAULT_TABLE table=test-table [id=b7fd20122fdc4d99b0f3e426f53cab87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:15.358438 6619 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca2d0474e8146f3a6c51d71fc31842d (DEFAULT_TABLE table=test-table [id=b7fd20122fdc4d99b0f3e426f53cab87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:15.358973 6485 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca2d0474e8146f3a6c51d71fc31842d (DEFAULT_TABLE table=test-table [id=b7fd20122fdc4d99b0f3e426f53cab87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:15.359416 6351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca2d0474e8146f3a6c51d71fc31842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:15.359488 6619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca2d0474e8146f3a6c51d71fc31842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:15.359839 6485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca2d0474e8146f3a6c51d71fc31842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:15.366671 6702 tablet_bootstrap.cc:492] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc: Bootstrap starting.
I20260929 10:27:15.367091 6703 tablet_bootstrap.cc:492] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Bootstrap starting.
I20260929 10:27:15.367112 6704 tablet_bootstrap.cc:492] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Bootstrap starting.
I20260929 10:27:15.368960 6704 tablet_bootstrap.cc:654] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:15.368960 6703 tablet_bootstrap.cc:654] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:15.370221 6703 log.cc:824] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:15.370303 6704 log.cc:824] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:15.371452 6702 tablet_bootstrap.cc:654] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:15.372251 6703 tablet_bootstrap.cc:492] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: No bootstrap required, opened a new log
I20260929 10:27:15.372440 6704 tablet_bootstrap.cc:492] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: No bootstrap required, opened a new log
I20260929 10:27:15.372473 6703 ts_tablet_manager.cc:1397] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 10:27:15.372637 6704 ts_tablet_manager.cc:1397] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:27:15.372733 6702 log.cc:824] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:15.374825 6702 tablet_bootstrap.cc:492] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc: No bootstrap required, opened a new log
I20260929 10:27:15.375002 6702 ts_tablet_manager.cc:1397] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 10:27:15.375675 6703 raft_consensus.cc:348] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.375965 6703 raft_consensus.cc:374] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:15.376017 6703 raft_consensus.cc:729] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40a928cd53240fb9ad160a89840ed91, State: Initialized, Role: FOLLOWER
I20260929 10:27:15.376471 6704 raft_consensus.cc:348] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.376761 6704 raft_consensus.cc:374] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:15.376827 6704 raft_consensus.cc:729] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6133a81473941d4901b3f7fbaca9537, State: Initialized, Role: FOLLOWER
I20260929 10:27:15.376714 6703 consensus_queue.cc:260] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.377274 6704 consensus_queue.cc:260] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.377583 6551 heartbeater.cc:503] Master 127.21.33.190:39781 was elected leader, sending a full tablet report...
I20260929 10:27:15.377947 6703 ts_tablet_manager.cc:1428] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:27:15.378393 6704 ts_tablet_manager.cc:1428] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:27:15.378714 6685 heartbeater.cc:503] Master 127.21.33.190:39781 was elected leader, sending a full tablet report...
I20260929 10:27:15.378819 6702 raft_consensus.cc:348] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.379050 6702 raft_consensus.cc:374] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:15.379097 6702 raft_consensus.cc:729] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73a5a5c10a74c5597c4c2f368b547dc, State: Initialized, Role: FOLLOWER
I20260929 10:27:15.379495 6702 consensus_queue.cc:260] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.380276 6417 heartbeater.cc:503] Master 127.21.33.190:39781 was elected leader, sending a full tablet report...
I20260929 10:27:15.380457 6702 ts_tablet_manager.cc:1428] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:27:15.522079 6710 raft_consensus.cc:482] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:15.522310 6710 raft_consensus.cc:504] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.523479 6710 leader_election.cc:288] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6133a81473941d4901b3f7fbaca9537 (127.21.33.131:45351), d40a928cd53240fb9ad160a89840ed91 (127.21.33.130:33467)
I20260929 10:27:15.527833 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.523886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:37383 (local address 127.21.33.131:45351)
0929 10:27:15.524023 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.524027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.524257 (+ 230us) server_negotiation.cc:407] Connection header received
0929 10:27:15.524428 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.524433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.524490 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.524596 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:15.524948 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.525565 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.526784 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.527266 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.527357 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.527434 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.527636 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.527702 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 10:27:15.527875 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.523829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:45351 (local address 127.21.33.129:37383)
0929 10:27:15.524151 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.524169 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:15.524299 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:15.524604 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:15.524607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:15.524648 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:15.524838 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.524845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.525668 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.525671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.526663 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.526673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.527396 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.527400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.527484 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.527512 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:27:15.527582 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 10:27:15.527666 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":46,"thread_start_us":111,"threads_started":1}
I20260929 10:27:15.527902 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.524376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:41225 (local address 127.21.33.130:33467)
0929 10:27:15.524512 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.524515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.524661 (+ 146us) server_negotiation.cc:407] Connection header received
0929 10:27:15.524779 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.524782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.524834 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.524919 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:15.525172 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.525781 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.526833 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.527291 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.527428 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.527482 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.527715 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.527794 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:27:15.527882 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.524230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33467 (local address 127.21.33.129:41225)
0929 10:27:15.524567 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.524583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:15.524679 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:15.524946 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:15.524949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:15.524965 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:15.525068 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.525074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.525887 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.525890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.526692 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:15.526701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.527434 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.527439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:15.527532 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.527548 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 10:27:15.527639 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 10:27:15.527705 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260929 10:27:15.528635 6505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca2d0474e8146f3a6c51d71fc31842d" candidate_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40a928cd53240fb9ad160a89840ed91" is_pre_election: true
I20260929 10:27:15.528637 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca2d0474e8146f3a6c51d71fc31842d" candidate_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6133a81473941d4901b3f7fbaca9537" is_pre_election: true
I20260929 10:27:15.528920 6505 raft_consensus.cc:2454] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73a5a5c10a74c5597c4c2f368b547dc in term 0.
I20260929 10:27:15.528939 6639 raft_consensus.cc:2454] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73a5a5c10a74c5597c4c2f368b547dc in term 0.
I20260929 10:27:15.529500 6306 leader_election.cc:302] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b73a5a5c10a74c5597c4c2f368b547dc, f6133a81473941d4901b3f7fbaca9537; no voters:
I20260929 10:27:15.529762 6710 raft_consensus.cc:2781] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:27:15.529872 6710 raft_consensus.cc:482] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:27:15.529929 6710 raft_consensus.cc:3037] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:15.531054 6710 raft_consensus.cc:504] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.531394 6710 leader_election.cc:288] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [CANDIDATE]: Term 1 election: Requested vote from peers f6133a81473941d4901b3f7fbaca9537 (127.21.33.131:45351), d40a928cd53240fb9ad160a89840ed91 (127.21.33.130:33467)
I20260929 10:27:15.531772 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca2d0474e8146f3a6c51d71fc31842d" candidate_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6133a81473941d4901b3f7fbaca9537"
I20260929 10:27:15.531884 6505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca2d0474e8146f3a6c51d71fc31842d" candidate_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d40a928cd53240fb9ad160a89840ed91"
I20260929 10:27:15.531913 6639 raft_consensus.cc:3037] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:15.532049 6505 raft_consensus.cc:3037] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:15.532964 6639 raft_consensus.cc:2454] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73a5a5c10a74c5597c4c2f368b547dc in term 1.
I20260929 10:27:15.533308 6505 raft_consensus.cc:2454] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73a5a5c10a74c5597c4c2f368b547dc in term 1.
I20260929 10:27:15.533322 6306 leader_election.cc:302] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b73a5a5c10a74c5597c4c2f368b547dc, f6133a81473941d4901b3f7fbaca9537; no voters:
I20260929 10:27:15.533499 6710 raft_consensus.cc:2781] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:15.533731 6710 raft_consensus.cc:686] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [term 1 LEADER]: Becoming Leader. State: Replica: b73a5a5c10a74c5597c4c2f368b547dc, State: Running, Role: LEADER
I20260929 10:27:15.534103 6710 consensus_queue.cc:237] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } }
I20260929 10:27:15.537364 6226 catalog_manager.cc:5495] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc reported cstate change: term changed from 0 to 1, leader changed from <none> to b73a5a5c10a74c5597c4c2f368b547dc (127.21.33.129). New cstate: current_term: 1 leader_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73a5a5c10a74c5597c4c2f368b547dc" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 44225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 } health_report { overall_health: UNKNOWN } } }
W20260929 10:27:15.548880 6686 tablet.cc:2369] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:27:15.549065 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.545904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.21.33.129:44225)
0929 10:27:15.546032 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.546036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.546049 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:27:15.546167 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.546170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.546220 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.546303 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:15.546511 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.546997 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.547760 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.547963 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.548041 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.548282 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:15.548714 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:15.548846 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.548882 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.548939 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:27:15.554558 6639 raft_consensus.cc:1261] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 1 FOLLOWER]: Refusing update from remote peer b73a5a5c10a74c5597c4c2f368b547dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:15.554559 6505 raft_consensus.cc:1261] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 1 FOLLOWER]: Refusing update from remote peer b73a5a5c10a74c5597c4c2f368b547dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:15.555243 6710 consensus_queue.cc:1035] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6133a81473941d4901b3f7fbaca9537" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 45351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:15.555444 6713 consensus_queue.cc:1035] T 5ca2d0474e8146f3a6c51d71fc31842d P b73a5a5c10a74c5597c4c2f368b547dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d40a928cd53240fb9ad160a89840ed91" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:15.571102 6225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:15.571308 6225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:15.574041 6225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a937aeef8fd4e7fb4708a3b5cfdbdba: Sending DeleteTablet for 3 replicas of tablet 5ca2d0474e8146f3a6c51d71fc31842d
I20260929 10:27:15.574817 6351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca2d0474e8146f3a6c51d71fc31842d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:52440
I20260929 10:27:15.575217 6619 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca2d0474e8146f3a6c51d71fc31842d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:56332
I20260929 10:27:15.575666 6485 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca2d0474e8146f3a6c51d71fc31842d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:15 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:55552
I20260929 10:27:15.576334 6727 tablet_replica.cc:330] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: stopping tablet replica
I20260929 10:27:15.576645 6727 raft_consensus.cc:2229] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:15.576932 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 6288
I20260929 10:27:15.576951 6727 raft_consensus.cc:2258] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:15.578289 6728 tablet_replica.cc:330] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: stopping tablet replica
I20260929 10:27:15.578585 6728 raft_consensus.cc:2229] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:15.578891 6728 raft_consensus.cc:2258] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:15.579983 6727 ts_tablet_manager.cc:1905] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:15.580169 6728 ts_tablet_manager.cc:1905] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:15.582726 6727 ts_tablet_manager.cc:1918] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:15.582726 6728 ts_tablet_manager.cc:1918] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:15.582827 6728 log.cc:1196] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/5ca2d0474e8146f3a6c51d71fc31842d
I20260929 10:27:15.582827 6727 log.cc:1196] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/5ca2d0474e8146f3a6c51d71fc31842d
I20260929 10:27:15.583149 6727 ts_tablet_manager.cc:1939] T 5ca2d0474e8146f3a6c51d71fc31842d P f6133a81473941d4901b3f7fbaca9537: Deleting consensus metadata
I20260929 10:27:15.583158 6728 ts_tablet_manager.cc:1939] T 5ca2d0474e8146f3a6c51d71fc31842d P d40a928cd53240fb9ad160a89840ed91: Deleting consensus metadata
I20260929 10:27:15.584398 6212 catalog_manager.cc:4841] TS d40a928cd53240fb9ad160a89840ed91 (127.21.33.130:33467): tablet 5ca2d0474e8146f3a6c51d71fc31842d (table test-table [id=b7fd20122fdc4d99b0f3e426f53cab87]) successfully deleted
I20260929 10:27:15.584620 6212 catalog_manager.cc:4841] TS f6133a81473941d4901b3f7fbaca9537 (127.21.33.131:45351): tablet 5ca2d0474e8146f3a6c51d71fc31842d (table test-table [id=b7fd20122fdc4d99b0f3e426f53cab87]) successfully deleted
W20260929 10:27:15.588812 6210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:44225 (error 108)
I20260929 10:27:15.588852 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 6422
I20260929 10:27:15.590034 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.589673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:44225 (local address 127.0.0.1:52456)
0929 10:27:15.589865 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:15.589937 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 10:27:15.590207 6210 catalog_manager.cc:4584] TS b73a5a5c10a74c5597c4c2f368b547dc (127.21.33.129:44225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ca2d0474e8146f3a6c51d71fc31842d: Network error: Client connection negotiation failed: client connection to 127.21.33.129:44225: connect: Connection refused (error 111)
I20260929 10:27:15.595144 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 6556
I20260929 10:27:15.601336 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 6194
2026-09-29T10:27:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2390 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T10:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:15Z Disabled control of system clock
I20260929 10:27:15.631839 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:36177
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:27:15.740653 6733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:15.740913 6733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:15.740948 6733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:15.744833 6733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:15.744886 6733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:15.744902 6733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:15.744918 6733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:15.744933 6733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:15.749643 6733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:36177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:36177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:15.750643 6733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:15.751945 6733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:15.760512 6739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:15.760584 6733 server_base.cc:1034] running on GCE node
W20260929 10:27:15.760512 6738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:15.760533 6741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:15.761327 6733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:15.762491 6733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:15.763676 6733 hybrid_clock.cc:648] HybridClock initialized: now 1790677635763662 us; error 26 us; skew 500 ppm
I20260929 10:27:15.765589 6733 webserver.cc:460] Webserver started at http://127.21.33.190:42953/ using document root <none> and password file <none>
I20260929 10:27:15.766261 6733 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:15.766356 6733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:15.766590 6733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:15.768622 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "f836b07a16104326bd32c7330416ce55"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "54ea3d8464efc17e3da5eb035f099edc"
server_key_iv: "69430a8476d3beae35cb2879a40739bc"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.769232 6733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "f836b07a16104326bd32c7330416ce55"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "54ea3d8464efc17e3da5eb035f099edc"
server_key_iv: "69430a8476d3beae35cb2879a40739bc"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.772796 6733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:27:15.775239 6747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:15.776506 6733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:15.776665 6733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "f836b07a16104326bd32c7330416ce55"
format_stamp: "Formatted at 2026-09-29 10:27:15 on dist-test-slave-2rbl"
server_key: "54ea3d8464efc17e3da5eb035f099edc"
server_key_iv: "69430a8476d3beae35cb2879a40739bc"
server_key_version: "encryptionkey@0"
I20260929 10:27:15.776824 6733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:15.787437 6733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:15.788077 6733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:15.788221 6733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:15.795516 6733 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:36177
I20260929 10:27:15.795542 6799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:36177 every 8 connection(s)
I20260929 10:27:15.796478 6733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:15.797596 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 6733
I20260929 10:27:15.797753 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:15.798031 21638 external_mini_cluster.cc:1428] Setting key 7ec017ae4ec5eb54178fc1297523b4f6
I20260929 10:27:15.800244 6800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:15.806584 6800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55: Bootstrap starting.
I20260929 10:27:15.807305 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:15.799391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.21.33.190:36177)
0929 10:27:15.800002 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:15.800021 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:15.800057 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:27:15.800737 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:15.800753 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:15.801131 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:15.801471 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:15.801807 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.802756 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.803945 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:15.804397 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:15.804511 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:15.805453 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:15.805458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:15.805470 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:15.805862 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:15.805918 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:15.805937 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:15.806013 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:15.806224 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 10:27:15.806534 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":149,"threads_started":1}
I20260929 10:27:15.808805 6800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:15.809700 6800 log.cc:824] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:15.812084 6800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55: No bootstrap required, opened a new log
I20260929 10:27:15.817878 6800 raft_consensus.cc:348] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f836b07a16104326bd32c7330416ce55" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36177 } }
I20260929 10:27:15.818245 6800 raft_consensus.cc:374] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:15.818310 6800 raft_consensus.cc:729] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f836b07a16104326bd32c7330416ce55, State: Initialized, Role: FOLLOWER
I20260929 10:27:15.818841 6800 consensus_queue.cc:260] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f836b07a16104326bd32c7330416ce55" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36177 } }
I20260929 10:27:15.819015 6800 raft_consensus.cc:388] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:15.819089 6800 raft_consensus.cc:482] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:15.819185 6800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:15.820346 6800 raft_consensus.cc:504] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f836b07a16104326bd32c7330416ce55" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36177 } }
I20260929 10:27:15.820761 6800 leader_election.cc:302] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f836b07a16104326bd32c7330416ce55; no voters:
I20260929 10:27:15.821143 6800 leader_election.cc:288] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:15.821535 6805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:15.821952 6805 raft_consensus.cc:686] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [term 1 LEADER]: Becoming Leader. State: Replica: f836b07a16104326bd32c7330416ce55, State: Running, Role: LEADER
I20260929 10:27:15.822261 6805 consensus_queue.cc:237] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f836b07a16104326bd32c7330416ce55" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36177 } }
I20260929 10:27:15.822741 6800 sys_catalog.cc:564] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:15.824046 6806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f836b07a16104326bd32c7330416ce55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f836b07a16104326bd32c7330416ce55" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36177 } } }
I20260929 10:27:15.824168 6806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:15.824725 6807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f836b07a16104326bd32c7330416ce55. Latest consensus state: current_term: 1 leader_uuid: "f836b07a16104326bd32c7330416ce55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f836b07a16104326bd32c7330416ce55" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 36177 } } }
I20260929 10:27:15.824812 6807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:15.824855 6814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:15.828728 6814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:15.835523 6814 catalog_manager.cc:1347] Generated new cluster ID: fdbcaa0b9fa2457784f7a52bbc25ef6f
I20260929 10:27:15.835667 6814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:15.884850 6814 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:15.886059 6814 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:15.890573 6814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55: Generated new TSK 0
I20260929 10:27:15.891243 6814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:15.901808 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36177
--builtin_ntp_servers=127.21.33.148:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:27:16.020252 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:16.020530 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:16.020570 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:16.025151 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:16.025243 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:16.025303 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:27:16.030696 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:16.031731 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:16.033075 6824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:16.042438 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:16.042474 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:16.042474 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:16.042696 6824 server_base.cc:1034] running on GCE node
I20260929 10:27:16.043126 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:16.043758 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:16.044960 6824 hybrid_clock.cc:648] HybridClock initialized: now 1790677636044935 us; error 37 us; skew 500 ppm
I20260929 10:27:16.047049 6824 webserver.cc:460] Webserver started at http://127.21.33.129:39585/ using document root <none> and password file <none>
I20260929 10:27:16.047621 6824 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:16.047734 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:16.048043 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:16.050467 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "2cbeff5f45ec43d8ba351e718af55ae6"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "d8a785a6a1a2603e562efae967928bec"
server_key_iv: "78959d3b9ee9fc4b7857360d3b7560b8"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.051059 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "2cbeff5f45ec43d8ba351e718af55ae6"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "d8a785a6a1a2603e562efae967928bec"
server_key_iv: "78959d3b9ee9fc4b7857360d3b7560b8"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.054862 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:27:16.057616 6838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.058986 6824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:27:16.059118 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "2cbeff5f45ec43d8ba351e718af55ae6"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "d8a785a6a1a2603e562efae967928bec"
server_key_iv: "78959d3b9ee9fc4b7857360d3b7560b8"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.059242 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:16.071552 6824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:16.072283 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:16.072499 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:16.073122 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:16.074112 6824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:16.074183 6824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.074254 6824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:16.074304 6824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.083354 6824 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:40303
I20260929 10:27:16.083379 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:40303 every 8 connection(s)
I20260929 10:27:16.084236 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:27:16.089079 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 6824
I20260929 10:27:16.089238 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:27:16.089540 21638 external_mini_cluster.cc:1428] Setting key f28daf8c8b884a147c04d0c34db8a1c6
I20260929 10:27:16.091698 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36177
--builtin_ntp_servers=127.21.33.148:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:27:16.094684 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.086026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:34355 (local address 127.21.33.190:36177)
0929 10:27:16.086176 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.086181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.087313 (+ 1132us) server_negotiation.cc:407] Connection header received
0929 10:27:16.088671 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.088673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.088722 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.088814 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:16.090069 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.090826 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.092600 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.092820 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.092943 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.094157 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:16.094161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:16.094164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:16.094183 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:16.094201 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:16.094208 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:16.094286 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.094457 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.094536 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:27:16.095494 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.086494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36177 (local address 127.21.33.129:34355)
0929 10:27:16.087168 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.087208 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.088478 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.089049 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.089060 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.089180 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:16.089904 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.089925 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.091023 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.091027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.092467 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.092482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.092984 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.092988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.093181 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.093882 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:16.093883 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:16.093924 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:16.093989 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:16.094303 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:16.094314 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:16.094326 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.094453 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.094731 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260929 10:27:16.096712 6952 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36177
I20260929 10:27:16.096959 6952 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:16.097519 6952 heartbeater.cc:511] Master 127.21.33.190:36177 requested a full tablet report, sending...
I20260929 10:27:16.098999 6764 ts_manager.cc:194] Registered new tserver with Master: 2cbeff5f45ec43d8ba351e718af55ae6 (127.21.33.129:40303)
I20260929 10:27:16.100116 6764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.129:34355
W20260929 10:27:16.203701 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:16.203994 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:16.204066 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:16.208225 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:16.208328 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:16.208417 6955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:27:16.213301 6955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:16.214418 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:16.215837 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:16.225962 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:16.225984 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:16.225965 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:16.226058 6955 server_base.cc:1034] running on GCE node
I20260929 10:27:16.226606 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:16.227253 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:16.228406 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790677636228391 us; error 27 us; skew 500 ppm
I20260929 10:27:16.230296 6955 webserver.cc:460] Webserver started at http://127.21.33.130:39031/ using document root <none> and password file <none>
I20260929 10:27:16.230857 6955 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:16.230918 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:16.231113 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:16.233117 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "da6258223e564606ba03bac339b8559c"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "be60b55df908c0b3ee3d43cc7e881acd"
server_key_iv: "02c242e1af7245a9f72a0a490d99947f"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.233711 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "da6258223e564606ba03bac339b8559c"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "be60b55df908c0b3ee3d43cc7e881acd"
server_key_iv: "02c242e1af7245a9f72a0a490d99947f"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.237146 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:27:16.239625 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.240823 6955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:16.240912 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "da6258223e564606ba03bac339b8559c"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "be60b55df908c0b3ee3d43cc7e881acd"
server_key_iv: "02c242e1af7245a9f72a0a490d99947f"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.240993 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:16.270737 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:16.271430 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:16.271582 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:16.272228 6955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:16.273252 6955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:16.273299 6955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.273339 6955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:16.273355 6955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.282774 6955 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:45175
I20260929 10:27:16.282801 7082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:45175 every 8 connection(s)
I20260929 10:27:16.283689 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:27:16.289161 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 6955
I20260929 10:27:16.289335 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:27:16.289652 21638 external_mini_cluster.cc:1428] Setting key 944a9f77d322ea99c41769e654a230e7
I20260929 10:27:16.291563 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36177
--builtin_ntp_servers=127.21.33.148:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:27:16.293440 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.285561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:56019 (local address 127.21.33.190:36177)
0929 10:27:16.285719 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.285722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.286728 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 10:27:16.288134 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.288138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.288195 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.288298 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:16.289232 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.289902 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.291216 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.291445 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.291554 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.292807 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:16.292814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:16.292818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:16.292838 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:16.292865 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:16.292877 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:16.292960 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.293142 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.293267 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 10:27:16.294580 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.285931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36177 (local address 127.21.33.130:56019)
0929 10:27:16.286568 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.286616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.287914 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.288469 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.288476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.288559 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:16.289055 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.289068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.290028 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.290036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.291079 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.291089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.291594 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.291598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.291841 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.292487 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:16.292489 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:16.292532 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:16.292598 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:16.292976 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:16.292986 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:16.292994 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.293137 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.293495 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260929 10:27:16.295874 7083 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36177
I20260929 10:27:16.296128 7083 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:16.296697 7083 heartbeater.cc:511] Master 127.21.33.190:36177 requested a full tablet report, sending...
I20260929 10:27:16.297808 6764 ts_manager.cc:194] Registered new tserver with Master: da6258223e564606ba03bac339b8559c (127.21.33.130:45175)
I20260929 10:27:16.298409 6764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.130:56019
W20260929 10:27:16.402218 7086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:16.402500 7086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:16.402540 7086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:16.406562 7086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:16.406627 7086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:16.406683 7086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:27:16.411360 7086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:36177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:16.412375 7086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:16.413736 7086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:16.424460 7094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:16.424484 7092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:16.424515 7091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:16.424970 7086 server_base.cc:1034] running on GCE node
I20260929 10:27:16.425467 7086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:16.426218 7086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:16.427443 7086 hybrid_clock.cc:648] HybridClock initialized: now 1790677636427422 us; error 31 us; skew 500 ppm
I20260929 10:27:16.429579 7086 webserver.cc:460] Webserver started at http://127.21.33.131:37181/ using document root <none> and password file <none>
I20260929 10:27:16.430228 7086 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:16.430332 7086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:16.430589 7086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:16.432910 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "0b829d3be2394e04903435d7f129a42e"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "243c27bdb0cbd2728836c574875a7e10"
server_key_iv: "cda40d6f656b272c7825d7a5b1407fcf"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.433606 7086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "0b829d3be2394e04903435d7f129a42e"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "243c27bdb0cbd2728836c574875a7e10"
server_key_iv: "cda40d6f656b272c7825d7a5b1407fcf"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.437661 7086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:27:16.440346 7100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.441695 7086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:16.441844 7086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "0b829d3be2394e04903435d7f129a42e"
format_stamp: "Formatted at 2026-09-29 10:27:16 on dist-test-slave-2rbl"
server_key: "243c27bdb0cbd2728836c574875a7e10"
server_key_iv: "cda40d6f656b272c7825d7a5b1407fcf"
server_key_version: "encryptionkey@0"
I20260929 10:27:16.441972 7086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:16.453287 7086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:16.453998 7086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:16.454241 7086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:16.454882 7086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:16.455905 7086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:16.455988 7086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.456067 7086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:16.456121 7086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:16.466924 7086 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:35599
I20260929 10:27:16.466944 7213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:35599 every 8 connection(s)
I20260929 10:27:16.467947 7086 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:27:16.468600 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 7086
I20260929 10:27:16.468748 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:27:16.469166 21638 external_mini_cluster.cc:1428] Setting key 0e160d979ae1f858a21cef5ead70543a
I20260929 10:27:16.477568 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.470024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:49013 (local address 127.21.33.190:36177)
0929 10:27:16.470288 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.470294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.471559 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 10:27:16.472939 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.472942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.472986 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.473054 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:16.474023 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.474545 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.475729 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.475939 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.476018 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.477073 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:16.477076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:16.477078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:16.477090 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:16.477107 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:16.477114 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:16.477171 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.477404 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.477454 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260929 10:27:16.478487 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.470675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:36177 (local address 127.21.33.131:49013)
0929 10:27:16.471410 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.471460 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.472761 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.473333 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.473340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.473425 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:16.473908 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.473920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.474670 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.474673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.475612 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.475622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.476089 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.476092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.476241 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.476825 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:16.476826 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:16.476865 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:16.476930 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:16.477191 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:16.477264 (+ 73us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:16.477273 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.477394 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.477619 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":146,"threads_started":1}
I20260929 10:27:16.479599 7214 heartbeater.cc:347] Connected to a master server at 127.21.33.190:36177
I20260929 10:27:16.479843 7214 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:16.480379 7214 heartbeater.cc:511] Master 127.21.33.190:36177 requested a full tablet report, sending...
I20260929 10:27:16.481480 6764 ts_manager.cc:194] Registered new tserver with Master: 0b829d3be2394e04903435d7f129a42e (127.21.33.131:35599)
I20260929 10:27:16.482163 6764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.131:49013
I20260929 10:27:16.483126 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:27:16.488782 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.485022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.21.33.190:36177)
0929 10:27:16.485163 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.485167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.485394 (+ 227us) server_negotiation.cc:407] Connection header received
0929 10:27:16.485472 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.485474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.485511 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.485571 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:16.485794 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.486313 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.487311 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.487523 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.487625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.488347 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:16.488350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:16.488352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:16.488363 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:16.488381 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:16.488388 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:16.488451 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.488524 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.488578 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:27:16.491503 6764 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:27:16.494627 6764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:27:16.508185 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.502553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:45175 (local address 127.0.0.1:53458)
0929 10:27:16.502808 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.502847 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.502978 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.504142 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.504147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.504172 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:16.504293 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.504298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.505513 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.505518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.506722 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.506731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.507638 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.507642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.507744 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.507783 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.507893 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.508020 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"spinlock_wait_cycles":7296,"thread_start_us":97,"threads_started":1}
I20260929 10:27:16.508201 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.503337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.21.33.130:45175)
0929 10:27:16.503746 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.503757 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.503776 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:27:16.503838 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.503841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.503978 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.504130 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:16.504411 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.505410 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.506860 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.507526 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.507639 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.507826 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.507941 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.508034 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":105,"threads_started":1}
I20260929 10:27:16.508734 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.502470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:40303 (local address 127.0.0.1:51382)
0929 10:27:16.502800 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.502846 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.502978 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.504393 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.504395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.504407 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:16.504509 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.504518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.505969 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.505972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.507049 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.507058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.508298 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.508303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.508418 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.508444 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.508517 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.508605 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"spinlock_wait_cycles":2816,"thread_start_us":92,"threads_started":1}
I20260929 10:27:16.508985 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.503194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:35599 (local address 127.0.0.1:45878)
0929 10:27:16.504371 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.504385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.504485 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.504925 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.504928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.504946 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:16.505032 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.505038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.506336 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.506340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.507391 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.507401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.508621 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.508624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.508710 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.508732 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.508804 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.508868 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"thread_start_us":94,"threads_started":1}
I20260929 10:27:16.509287 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.503180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.21.33.129:40303)
0929 10:27:16.503635 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.503648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.503674 (+ 26us) server_negotiation.cc:407] Connection header received
0929 10:27:16.503755 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.503760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.503961 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.504139 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:16.504647 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.505848 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.507181 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.507969 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.508062 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.508230 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.508974 (+ 744us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.509086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":140,"threads_started":1}
I20260929 10:27:16.510201 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.503344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.21.33.131:35599)
0929 10:27:16.503931 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.503952 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.504493 (+ 541us) server_negotiation.cc:407] Connection header received
0929 10:27:16.504594 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.504602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.504784 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.504929 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:16.505144 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.506208 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.507524 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.508304 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.508402 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.508558 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.509248 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.510055 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":124,"threads_started":1}
I20260929 10:27:16.511575 7017 tablet_service.cc:1467] Processing CreateTablet for tablet 478fb43b554a4b8796d8743c62805647 (DEFAULT_TABLE table=test-table [id=de061c168aaf4e0d9c9484f4c4c739ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:16.512013 6886 tablet_service.cc:1467] Processing CreateTablet for tablet 478fb43b554a4b8796d8743c62805647 (DEFAULT_TABLE table=test-table [id=de061c168aaf4e0d9c9484f4c4c739ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:16.512259 7148 tablet_service.cc:1467] Processing CreateTablet for tablet 478fb43b554a4b8796d8743c62805647 (DEFAULT_TABLE table=test-table [id=de061c168aaf4e0d9c9484f4c4c739ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:16.512742 7017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478fb43b554a4b8796d8743c62805647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:16.512997 6886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478fb43b554a4b8796d8743c62805647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:16.513188 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 478fb43b554a4b8796d8743c62805647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:16.519960 7231 tablet_bootstrap.cc:492] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Bootstrap starting.
I20260929 10:27:16.520502 7233 tablet_bootstrap.cc:492] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Bootstrap starting.
I20260929 10:27:16.521543 7232 tablet_bootstrap.cc:492] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6: Bootstrap starting.
I20260929 10:27:16.522233 7231 tablet_bootstrap.cc:654] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:16.522502 7233 tablet_bootstrap.cc:654] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:16.523037 7231 log.cc:824] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:16.523329 7233 log.cc:824] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:16.523600 7232 tablet_bootstrap.cc:654] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:16.524371 7232 log.cc:824] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:16.525044 7231 tablet_bootstrap.cc:492] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: No bootstrap required, opened a new log
I20260929 10:27:16.525254 7231 ts_tablet_manager.cc:1397] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:27:16.525375 7233 tablet_bootstrap.cc:492] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: No bootstrap required, opened a new log
I20260929 10:27:16.525547 7233 ts_tablet_manager.cc:1397] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:27:16.526759 7232 tablet_bootstrap.cc:492] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6: No bootstrap required, opened a new log
I20260929 10:27:16.526944 7232 ts_tablet_manager.cc:1397] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:27:16.528530 7231 raft_consensus.cc:348] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.528774 7231 raft_consensus.cc:374] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:16.528833 7231 raft_consensus.cc:729] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b829d3be2394e04903435d7f129a42e, State: Initialized, Role: FOLLOWER
I20260929 10:27:16.529294 7231 consensus_queue.cc:260] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.529927 7214 heartbeater.cc:503] Master 127.21.33.190:36177 was elected leader, sending a full tablet report...
I20260929 10:27:16.529839 7233 raft_consensus.cc:348] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.530090 7233 raft_consensus.cc:374] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:16.530138 7233 raft_consensus.cc:729] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6258223e564606ba03bac339b8559c, State: Initialized, Role: FOLLOWER
I20260929 10:27:16.530280 7231 ts_tablet_manager.cc:1428] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:27:16.530539 7233 consensus_queue.cc:260] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.530658 7232 raft_consensus.cc:348] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.530921 7232 raft_consensus.cc:374] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:16.530982 7232 raft_consensus.cc:729] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbeff5f45ec43d8ba351e718af55ae6, State: Initialized, Role: FOLLOWER
I20260929 10:27:16.531370 7232 consensus_queue.cc:260] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.531479 7233 ts_tablet_manager.cc:1428] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:27:16.531808 7083 heartbeater.cc:503] Master 127.21.33.190:36177 was elected leader, sending a full tablet report...
I20260929 10:27:16.532166 7232 ts_tablet_manager.cc:1428] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:27:16.532469 6952 heartbeater.cc:503] Master 127.21.33.190:36177 was elected leader, sending a full tablet report...
W20260929 10:27:16.534605 7084 tablet.cc:2369] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:27:16.535280 7238 raft_consensus.cc:482] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:16.535424 7238 raft_consensus.cc:504] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.536542 7238 leader_election.cc:288] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cbeff5f45ec43d8ba351e718af55ae6 (127.21.33.129:40303), 0b829d3be2394e04903435d7f129a42e (127.21.33.131:35599)
I20260929 10:27:16.540431 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.536790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:34951 (local address 127.21.33.129:40303)
0929 10:27:16.536890 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.536894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.537442 (+ 548us) server_negotiation.cc:407] Connection header received
0929 10:27:16.537566 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.537569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.537609 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.537696 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:16.537956 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.538456 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.539495 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.539884 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.540006 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.540054 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.540260 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.540328 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 10:27:16.540445 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.536701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:40303 (local address 127.21.33.130:34951)
0929 10:27:16.537356 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.537368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.537482 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.537705 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.537708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.537730 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:16.537853 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.537858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.538562 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.538567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.539378 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.539386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.539998 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.540002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.540129 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.540157 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.540243 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.540306 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":13}
I20260929 10:27:16.541257 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478fb43b554a4b8796d8743c62805647" candidate_uuid: "da6258223e564606ba03bac339b8559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" is_pre_election: true
I20260929 10:27:16.541531 6906 raft_consensus.cc:2454] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6258223e564606ba03bac339b8559c in term 0.
I20260929 10:27:16.541824 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.536967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:35599 (local address 127.21.33.130:41485)
0929 10:27:16.537445 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.537466 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:16.537563 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:16.538222 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:16.538226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:16.538244 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:16.538387 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.538394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.539246 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.539254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.540128 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:16.540138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.541268 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.541272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:16.541399 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.541427 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:27:16.541546 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 10:27:16.541621 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":6,"thread_start_us":239,"threads_started":1}
I20260929 10:27:16.542018 6971 leader_election.cc:302] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbeff5f45ec43d8ba351e718af55ae6, da6258223e564606ba03bac339b8559c; no voters:
I20260929 10:27:16.542212 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.537817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:41485 (local address 127.21.33.131:35599)
0929 10:27:16.537948 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.537954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.537970 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:27:16.538034 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.538037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.538084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.538196 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:16.538512 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.539109 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.540279 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.540918 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.541093 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.541149 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.541980 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.542085 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:27:16.542325 7238 raft_consensus.cc:2781] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:27:16.542408 7238 raft_consensus.cc:482] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:27:16.542452 7238 raft_consensus.cc:3037] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:16.542891 7168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478fb43b554a4b8796d8743c62805647" candidate_uuid: "da6258223e564606ba03bac339b8559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b829d3be2394e04903435d7f129a42e" is_pre_election: true
I20260929 10:27:16.543179 7168 raft_consensus.cc:2454] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6258223e564606ba03bac339b8559c in term 0.
I20260929 10:27:16.543557 7238 raft_consensus.cc:504] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.543928 7238 leader_election.cc:288] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [CANDIDATE]: Term 1 election: Requested vote from peers 2cbeff5f45ec43d8ba351e718af55ae6 (127.21.33.129:40303), 0b829d3be2394e04903435d7f129a42e (127.21.33.131:35599)
I20260929 10:27:16.544301 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478fb43b554a4b8796d8743c62805647" candidate_uuid: "da6258223e564606ba03bac339b8559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbeff5f45ec43d8ba351e718af55ae6"
I20260929 10:27:16.544386 7168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "478fb43b554a4b8796d8743c62805647" candidate_uuid: "da6258223e564606ba03bac339b8559c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b829d3be2394e04903435d7f129a42e"
I20260929 10:27:16.544436 6906 raft_consensus.cc:3037] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:16.544538 7168 raft_consensus.cc:3037] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:16.545631 6906 raft_consensus.cc:2454] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6258223e564606ba03bac339b8559c in term 1.
I20260929 10:27:16.545748 7168 raft_consensus.cc:2454] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6258223e564606ba03bac339b8559c in term 1.
I20260929 10:27:16.545972 6971 leader_election.cc:302] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbeff5f45ec43d8ba351e718af55ae6, da6258223e564606ba03bac339b8559c; no voters:
I20260929 10:27:16.546133 7238 raft_consensus.cc:2781] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:16.546372 7238 raft_consensus.cc:686] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 1 LEADER]: Becoming Leader. State: Replica: da6258223e564606ba03bac339b8559c, State: Running, Role: LEADER
I20260929 10:27:16.546679 7238 consensus_queue.cc:237] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } }
I20260929 10:27:16.549744 6763 catalog_manager.cc:5495] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c reported cstate change: term changed from 0 to 1, leader changed from <none> to da6258223e564606ba03bac339b8559c (127.21.33.130). New cstate: current_term: 1 leader_uuid: "da6258223e564606ba03bac339b8559c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6258223e564606ba03bac339b8559c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 45175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 } health_report { overall_health: UNKNOWN } } }
I20260929 10:27:16.560122 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.556837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.21.33.130:45175)
0929 10:27:16.556965 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:16.556970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:16.556987 (+ 17us) server_negotiation.cc:407] Connection header received
0929 10:27:16.557123 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:16.557125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:16.557180 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:16.557320 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:16.557504 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.558004 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.558882 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:16.559088 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:16.559172 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:16.559411 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:16.559807 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:16.559905 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:16.559955 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 10:27:16.559999 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:27:16.565963 7168 raft_consensus.cc:1261] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 1 FOLLOWER]: Refusing update from remote peer da6258223e564606ba03bac339b8559c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:16.566012 6906 raft_consensus.cc:1261] T 478fb43b554a4b8796d8743c62805647 P 2cbeff5f45ec43d8ba351e718af55ae6 [term 1 FOLLOWER]: Refusing update from remote peer da6258223e564606ba03bac339b8559c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:16.566651 7238 consensus_queue.cc:1035] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b829d3be2394e04903435d7f129a42e" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 35599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:16.566849 7241 consensus_queue.cc:1035] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbeff5f45ec43d8ba351e718af55ae6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:16.575116 7245 mvcc.cc:205] Tried to move back new op lower bound from 7334615599366598656 to 7334615599297949696. Current Snapshot: MvccSnapshot[applied={T|T < 7334615599366598656}]
I20260929 10:27:16.583483 6762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:16.583668 6762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:16.586360 6762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f836b07a16104326bd32c7330416ce55: Sending DeleteTablet for 3 replicas of tablet 478fb43b554a4b8796d8743c62805647
I20260929 10:27:16.587131 7017 tablet_service.cc:1514] Processing DeleteTablet for tablet 478fb43b554a4b8796d8743c62805647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:16 UTC) from {username='slave'} at 127.0.0.1:53458
I20260929 10:27:16.587205 7148 tablet_service.cc:1514] Processing DeleteTablet for tablet 478fb43b554a4b8796d8743c62805647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:16 UTC) from {username='slave'} at 127.0.0.1:45878
I20260929 10:27:16.587209 6886 tablet_service.cc:1514] Processing DeleteTablet for tablet 478fb43b554a4b8796d8743c62805647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:16 UTC) from {username='slave'} at 127.0.0.1:51382
I20260929 10:27:16.587633 7254 tablet_replica.cc:330] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: stopping tablet replica
I20260929 10:27:16.587674 7256 tablet_replica.cc:330] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: stopping tablet replica
I20260929 10:27:16.587977 7256 raft_consensus.cc:2229] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:16.588001 7254 raft_consensus.cc:2229] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:27:16.588311 7256 raft_consensus.cc:2258] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:16.588388 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 6824
I20260929 10:27:16.589354 7254 raft_consensus.cc:2258] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:16.590883 7254 ts_tablet_manager.cc:1905] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:16.593393 7254 ts_tablet_manager.cc:1918] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:16.593497 7254 log.cc:1196] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/478fb43b554a4b8796d8743c62805647
I20260929 10:27:16.593858 7254 ts_tablet_manager.cc:1939] T 478fb43b554a4b8796d8743c62805647 P da6258223e564606ba03bac339b8559c: Deleting consensus metadata
I20260929 10:27:16.594475 7256 ts_tablet_manager.cc:1905] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:16.595103 6750 catalog_manager.cc:4841] TS da6258223e564606ba03bac339b8559c (127.21.33.130:45175): tablet 478fb43b554a4b8796d8743c62805647 (table test-table [id=de061c168aaf4e0d9c9484f4c4c739ca]) successfully deleted
W20260929 10:27:16.596635 6749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:40303 (error 108)
I20260929 10:27:16.597118 7256 ts_tablet_manager.cc:1918] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:16.597256 7256 log.cc:1196] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/478fb43b554a4b8796d8743c62805647
I20260929 10:27:16.597461 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 6955
I20260929 10:27:16.598690 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:16.598423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:40303 (local address 127.0.0.1:51390)
0929 10:27:16.598532 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:16.598617 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 10:27:16.599014 6749 catalog_manager.cc:4584] TS 2cbeff5f45ec43d8ba351e718af55ae6 (127.21.33.129:40303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 478fb43b554a4b8796d8743c62805647: Network error: Client connection negotiation failed: client connection to 127.21.33.129:40303: connect: Connection refused (error 111)
I20260929 10:27:16.599104 7256 ts_tablet_manager.cc:1939] T 478fb43b554a4b8796d8743c62805647 P 0b829d3be2394e04903435d7f129a42e: Deleting consensus metadata
I20260929 10:27:16.600459 6749 catalog_manager.cc:4841] TS 0b829d3be2394e04903435d7f129a42e (127.21.33.131:35599): tablet 478fb43b554a4b8796d8743c62805647 (table test-table [id=de061c168aaf4e0d9c9484f4c4c739ca]) successfully deleted
I20260929 10:27:16.603865 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 7086
I20260929 10:27:16.609419 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 6733
2026-09-29T10:27:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1007 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:16 dist-test-slave-2rbl krb5kdc[7260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:16 dist-test-slave-2rbl krb5kdc[7260](info): set up 2 sockets
Sep 29 10:27:16 dist-test-slave-2rbl krb5kdc[7260](info): commencing operation
krb5kdc: starting...
W20260929 10:27:17.653612 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:17 dist-test-slave-2rbl krb5kdc[7260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:17.776875 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:42407
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:42407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:17.885509 7273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:17.885769 7273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:17.885851 7273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:17.889810 7273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:17.889864 7273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:17.889883 7273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:17.889947 7273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:17.889976 7273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:17.895200 7273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:42407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:42407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:17.896240 7273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:17.897287 7273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:27:17.905606 7273 server_base.cc:1034] running on GCE node
W20260929 10:27:17.905624 7281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:17.905578 7279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:17.905608 7278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:17.906330 7273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:17.907404 7273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:17.908593 7273 hybrid_clock.cc:648] HybridClock initialized: now 1790677637908577 us; error 27 us; skew 500 ppm
Sep 29 10:27:17 dist-test-slave-2rbl krb5kdc[7260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:17.911294 7273 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:17.912169 7273 webserver.cc:460] Webserver started at http://127.21.33.190:34513/ using document root <none> and password file <none>
I20260929 10:27:17.912729 7273 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:17.912818 7273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:17.913048 7273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:17.915112 7273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd"
format_stamp: "Formatted at 2026-09-29 10:27:17 on dist-test-slave-2rbl"
server_key: "2c07fd070d0b96871ef1b0a6f27b6178"
server_key_iv: "f8d1d4c5301584425324cccdfb068fc0"
server_key_version: "encryptionkey@0"
I20260929 10:27:17.915680 7273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd"
format_stamp: "Formatted at 2026-09-29 10:27:17 on dist-test-slave-2rbl"
server_key: "2c07fd070d0b96871ef1b0a6f27b6178"
server_key_iv: "f8d1d4c5301584425324cccdfb068fc0"
server_key_version: "encryptionkey@0"
I20260929 10:27:17.919445 7273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:27:17.921919 7288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:17.923171 7273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:17.923316 7273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd"
format_stamp: "Formatted at 2026-09-29 10:27:17 on dist-test-slave-2rbl"
server_key: "2c07fd070d0b96871ef1b0a6f27b6178"
server_key_iv: "f8d1d4c5301584425324cccdfb068fc0"
server_key_version: "encryptionkey@0"
I20260929 10:27:17.923437 7273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:17.940742 7273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:17.944213 7273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:17.944423 7273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:17.951582 7273 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:42407
I20260929 10:27:17.951607 7340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:42407 every 8 connection(s)
I20260929 10:27:17.952558 7273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:17.952956 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 7273
I20260929 10:27:17.953092 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:17.953423 21638 external_mini_cluster.cc:1428] Setting key 062dd72d2721bcad34db9a8cd8514b52
I20260929 10:27:17.955722 7341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:17.961263 7341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd: Bootstrap starting.
Sep 29 10:27:17 dist-test-slave-2rbl krb5kdc[7260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:17.963855 7341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:17.964781 7341 log.cc:824] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:17.967258 7341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd: No bootstrap required, opened a new log
I20260929 10:27:17.968286 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:17.954879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.21.33.190:42407)
0929 10:27:17.955316 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:17.955331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:17.955368 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:27:17.956069 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:17.956087 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:17.956417 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:17.956767 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:17.957590 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:17.958617 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:17.959506 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:17.959983 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:17.960151 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:17.962983 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:17.962988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:17.963001 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:17.963029 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:17.965014 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:17.965571 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:17.965576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:17.965581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:17.965769 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:17.966071 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:17.966075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:17.966077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:17.966608 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:17.966810 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:17.967031 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 10:27:17.967336 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":142,"threads_started":1}
I20260929 10:27:17.971565 7341 raft_consensus.cc:348] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42407 } }
I20260929 10:27:17.971787 7341 raft_consensus.cc:374] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:17.971872 7341 raft_consensus.cc:729] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde7f40a1d3a44eabeb6de5ed7cb24dd, State: Initialized, Role: FOLLOWER
I20260929 10:27:17.972335 7341 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42407 } }
I20260929 10:27:17.972523 7341 raft_consensus.cc:388] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:17.972611 7341 raft_consensus.cc:482] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:17.972714 7341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:17.973704 7341 raft_consensus.cc:504] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42407 } }
I20260929 10:27:17.974009 7341 leader_election.cc:302] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde7f40a1d3a44eabeb6de5ed7cb24dd; no voters:
I20260929 10:27:17.974303 7341 leader_election.cc:288] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:17.974433 7346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:17.974697 7346 raft_consensus.cc:686] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [term 1 LEADER]: Becoming Leader. State: Replica: bde7f40a1d3a44eabeb6de5ed7cb24dd, State: Running, Role: LEADER
I20260929 10:27:17.975018 7346 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42407 } }
I20260929 10:27:17.975590 7341 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:17.976680 7347 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42407 } } }
I20260929 10:27:17.976830 7347 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:17.977129 7348 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde7f40a1d3a44eabeb6de5ed7cb24dd. Latest consensus state: current_term: 1 leader_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde7f40a1d3a44eabeb6de5ed7cb24dd" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42407 } } }
I20260929 10:27:17.977253 7348 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:17.977283 7355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:17.980531 7355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:17.986510 7355 catalog_manager.cc:1347] Generated new cluster ID: c683d6ddcbf34633b4f3a9af9d9077ac
I20260929 10:27:17.986593 7355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:18.006595 7355 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:18.007589 7355 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:18.012964 7355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd: Generated new TSK 0
I20260929 10:27:18.013664 7355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:27:18.045362 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42407
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:18.155855 7367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:18.156140 7367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:18.156190 7367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:18.160203 7367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:18.160271 7367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:18.160332 7367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:27:18.165868 7367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:18.166896 7367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:18.167891 7367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:18.177611 7375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:18.177660 7372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:18.177651 7373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:18.177977 7367 server_base.cc:1034] running on GCE node
I20260929 10:27:18.178329 7367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:18.178911 7367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:18.180068 7367 hybrid_clock.cc:648] HybridClock initialized: now 1790677638180043 us; error 37 us; skew 500 ppm
Sep 29 10:27:18 dist-test-slave-2rbl krb5kdc[7260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:18.182912 7367 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:27:18.183837 7367 webserver.cc:460] Webserver started at http://127.21.33.129:46649/ using document root <none> and password file <none>
I20260929 10:27:18.184424 7367 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:18.184532 7367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:18.184779 7367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:18.186985 7367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "c500a1d4983b4a6a97a009fadf039940"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "deced66bc74740301bfca6a117a1e8bf"
server_key_iv: "78b877c72ed59dfb0f1b15d126236d5e"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.187605 7367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "c500a1d4983b4a6a97a009fadf039940"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "deced66bc74740301bfca6a117a1e8bf"
server_key_iv: "78b877c72ed59dfb0f1b15d126236d5e"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.191484 7367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 10:27:18.193994 7382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.195216 7367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:18.195374 7367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "c500a1d4983b4a6a97a009fadf039940"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "deced66bc74740301bfca6a117a1e8bf"
server_key_iv: "78b877c72ed59dfb0f1b15d126236d5e"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.195494 7367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:18.217456 7367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:18.221140 7367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:18.221426 7367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:18.222096 7367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:18.223001 7367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:18.223091 7367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.223191 7367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:18.223248 7367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.233466 7367 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:46823
I20260929 10:27:18.233489 7495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:46823 every 8 connection(s)
I20260929 10:27:18.234387 7367 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:27:18.241933 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 7367
I20260929 10:27:18.242079 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:27:18.242439 21638 external_mini_cluster.cc:1428] Setting key f4e4fc41ed6d6a1a31d68c8b3d8bc295
Sep 29 10:27:18 dist-test-slave-2rbl krb5kdc[7260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:18.248709 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.236282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:44941 (local address 127.21.33.190:42407)
0929 10:27:18.236446 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.236450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.237359 (+ 909us) server_negotiation.cc:407] Connection header received
0929 10:27:18.238722 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.238725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.238798 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.238903 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:18.240054 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.240592 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.241841 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.242220 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.242320 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.245031 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:18.245034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:18.245036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:18.245059 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:18.246645 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.247167 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.247173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.247175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.247255 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.247594 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.247599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.247603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.247794 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:18.247903 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.248393 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.248521 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 10:27:18.249663 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.236679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:42407 (local address 127.21.33.129:44941)
0929 10:27:18.237152 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.237257 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.238505 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.239086 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.239092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.239430 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:18.239903 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.239917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.240718 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.240729 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.241700 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.241712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.242386 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.242390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.242601 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.243278 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:18.243304 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:18.244850 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:18.246778 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:18.246785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:18.246800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:18.247055 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:18.247348 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:18.247352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:18.247354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:18.247499 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:18.247900 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:18.247906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:18.248167 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.248355 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.248673 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260929 10:27:18.250941 7496 heartbeater.cc:347] Connected to a master server at 127.21.33.190:42407
I20260929 10:27:18.251227 7496 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:27:18.251819 7496 heartbeater.cc:511] Master 127.21.33.190:42407 requested a full tablet report, sending...
I20260929 10:27:18.253487 7305 ts_manager.cc:194] Registered new tserver with Master: c500a1d4983b4a6a97a009fadf039940 (127.21.33.129:46823)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:27:18.254853 7305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:44941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:27:18.271868 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42407
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:18.387939 7501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:18.388245 7501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:18.388293 7501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:18.392378 7501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:18.392463 7501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:18.392529 7501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:27:18.398253 7501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:18.399523 7501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:18.400678 7501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:18.411469 7506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:18.411469 7509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:18.411509 7507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:18.411667 7501 server_base.cc:1034] running on GCE node
I20260929 10:27:18.412266 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:18.412976 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:18.414218 7501 hybrid_clock.cc:648] HybridClock initialized: now 1790677638414187 us; error 52 us; skew 500 ppm
Sep 29 10:27:18 dist-test-slave-2rbl krb5kdc[7260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:18.420735 7501 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:27:18.421752 7501 webserver.cc:460] Webserver started at http://127.21.33.130:39319/ using document root <none> and password file <none>
I20260929 10:27:18.422369 7501 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:18.422466 7501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:18.422708 7501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:18.425168 7501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "ab6f546d9ad04c34b24f28b77b312f29"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "ced8f57915d53a92b4871ea44d59eae2"
server_key_iv: "b8b8284869b39f32c9355472cf2225d5"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.425947 7501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "ab6f546d9ad04c34b24f28b77b312f29"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "ced8f57915d53a92b4871ea44d59eae2"
server_key_iv: "b8b8284869b39f32c9355472cf2225d5"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.430477 7501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 10:27:18.432803 7516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.434032 7501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:18.434158 7501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "ab6f546d9ad04c34b24f28b77b312f29"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "ced8f57915d53a92b4871ea44d59eae2"
server_key_iv: "b8b8284869b39f32c9355472cf2225d5"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.434243 7501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:18.445525 7501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:18.448714 7501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:18.448864 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:18.449445 7501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:18.450217 7501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:18.450263 7501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.450300 7501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:18.450316 7501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.459125 7501 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:37551
I20260929 10:27:18.459178 7629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:37551 every 8 connection(s)
I20260929 10:27:18.460026 7501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:27:18.468775 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 7501
I20260929 10:27:18.468920 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:27:18.469259 21638 external_mini_cluster.cc:1428] Setting key e4f2df533fff10b89ead348e6773c0c8
Sep 29 10:27:18 dist-test-slave-2rbl krb5kdc[7260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:18.474052 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.461860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:56503 (local address 127.21.33.190:42407)
0929 10:27:18.462012 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.462017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.462889 (+ 872us) server_negotiation.cc:407] Connection header received
0929 10:27:18.464263 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.464265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.464311 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.464383 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:18.465645 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.466181 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.467349 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.467683 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.467764 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.470497 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:18.470501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:18.470504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:18.470538 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:18.471959 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.472477 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.472482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.472484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.472570 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.472935 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.472938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.472940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.473124 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:18.473253 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.473751 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.473888 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:27:18.474870 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.462274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:42407 (local address 127.21.33.130:56503)
0929 10:27:18.462735 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.462779 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.464065 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.464552 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.464559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.464904 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:18.465504 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.465522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.466309 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.466312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.467224 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.467234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.467815 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.467817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.467956 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.468508 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:18.468524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:18.470330 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:18.472083 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:18.472089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:18.472104 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:18.472371 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:18.472678 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:18.472681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:18.472684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:18.472830 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:18.473254 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:18.473262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:18.473515 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.473709 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.474037 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":97,"threads_started":1}
I20260929 10:27:18.476263 7630 heartbeater.cc:347] Connected to a master server at 127.21.33.190:42407
I20260929 10:27:18.476508 7630 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:18.477053 7630 heartbeater.cc:511] Master 127.21.33.190:42407 requested a full tablet report, sending...
I20260929 10:27:18.478269 7305 ts_manager.cc:194] Registered new tserver with Master: ab6f546d9ad04c34b24f28b77b312f29 (127.21.33.130:37551)
I20260929 10:27:18.479115 7305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:56503
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:27:18.497012 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42407
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:18.608811 7635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:18.609133 7635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:18.609275 7635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:18.613427 7635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:18.613524 7635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:18.613628 7635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:27:18.619119 7635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:42407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:18.620285 7635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:18.621392 7635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:18.631320 7640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:18.631335 7641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:18.631490 7635 server_base.cc:1034] running on GCE node
W20260929 10:27:18.631678 7643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:18.632184 7635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:18.632912 7635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:18.634155 7635 hybrid_clock.cc:648] HybridClock initialized: now 1790677638634128 us; error 37 us; skew 500 ppm
Sep 29 10:27:18 dist-test-slave-2rbl krb5kdc[7260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:18.637367 7635 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:27:18.638352 7635 webserver.cc:460] Webserver started at http://127.21.33.131:40507/ using document root <none> and password file <none>
I20260929 10:27:18.638984 7635 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:18.639048 7635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:18.639322 7635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:18.641413 7635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "8c4934b66acf4a8ab4e12a7bd9feef04"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "a92053d0857f52f7832e7462103669dd"
server_key_iv: "88d66ab550bb14cc2f05ef256103e252"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.641973 7635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "8c4934b66acf4a8ab4e12a7bd9feef04"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "a92053d0857f52f7832e7462103669dd"
server_key_iv: "88d66ab550bb14cc2f05ef256103e252"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.645767 7635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:27:18.648257 7650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.649529 7635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:18.649674 7635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "8c4934b66acf4a8ab4e12a7bd9feef04"
format_stamp: "Formatted at 2026-09-29 10:27:18 on dist-test-slave-2rbl"
server_key: "a92053d0857f52f7832e7462103669dd"
server_key_iv: "88d66ab550bb14cc2f05ef256103e252"
server_key_version: "encryptionkey@0"
I20260929 10:27:18.649791 7635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:18.666501 7635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:18.669651 7635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:18.669868 7635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:18.670506 7635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:18.671437 7635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:18.671509 7635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.671587 7635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:18.671623 7635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:18.681116 7635 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:34539
I20260929 10:27:18.681139 7763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:34539 every 8 connection(s)
I20260929 10:27:18.682058 7635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:27:18.683573 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 7635
I20260929 10:27:18.683717 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:27:18.684046 21638 external_mini_cluster.cc:1428] Setting key 830a79faaf5578dda9045e483a1c43f7
Sep 29 10:27:18 dist-test-slave-2rbl krb5kdc[7260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:18.696511 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.684044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:38309 (local address 127.21.33.190:42407)
0929 10:27:18.684190 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.684193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.685258 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 10:27:18.686813 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.686816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.686861 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.686931 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:18.688047 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.688618 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.689810 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.690021 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.690109 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.692627 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:18.692631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:18.692633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:18.692664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:18.694452 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.695044 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.695047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.695048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.695126 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.695427 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.695429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.695430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.695603 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:18.695694 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.696251 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.696380 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:27:18.697928 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.684521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:42407 (local address 127.21.33.131:38309)
0929 10:27:18.685066 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.685125 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.686625 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.687085 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.687092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.687449 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:18.687926 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.687939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.688727 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.688730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.689708 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.689718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.690147 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.690168 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.690308 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.690948 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:18.690964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:18.692456 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:18.694584 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:18.694609 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:18.694627 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:18.694951 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:18.695213 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:18.695215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:18.695217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:18.695347 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:18.695727 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:18.695733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:18.695971 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.696161 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.696508 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":142,"threads_started":1}
I20260929 10:27:18.699087 7764 heartbeater.cc:347] Connected to a master server at 127.21.33.190:42407
I20260929 10:27:18.699357 7764 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:18.699930 7764 heartbeater.cc:511] Master 127.21.33.190:42407 requested a full tablet report, sending...
I20260929 10:27:18.701004 7305 ts_manager.cc:194] Registered new tserver with Master: 8c4934b66acf4a8ab4e12a7bd9feef04 (127.21.33.131:34539)
I20260929 10:27:18.701653 7305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:38309
I20260929 10:27:18.710548 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:27:18.720548 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.712593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.21.33.190:42407)
0929 10:27:18.712753 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.712758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.712927 (+ 169us) server_negotiation.cc:407] Connection header received
0929 10:27:18.713016 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.713018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.713060 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.713150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:18.713929 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.714492 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.715331 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.715557 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.715648 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.717280 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:18.717283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:18.717285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:18.717311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:18.718832 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.719302 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.719305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.719306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.719380 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:18.719714 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:18.719716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:18.719717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:18.719928 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:18.720040 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.720268 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.720372 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 10:27:18.723718 7305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:27:18.726830 7305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:27:18.741719 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.735406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.21.33.131:34539)
0929 10:27:18.736613 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.736635 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.736845 (+ 210us) server_negotiation.cc:407] Connection header received
0929 10:27:18.736934 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.736940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.737105 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.737355 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:18.737762 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.738911 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.740287 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.741074 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.741176 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.741334 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.741461 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.741558 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":113,"threads_started":1}
I20260929 10:27:18.741788 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.735910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.21.33.129:46823)
0929 10:27:18.736285 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.736300 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.736324 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:27:18.736528 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.736538 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.736777 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.736989 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:18.737341 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.738354 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.740014 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.740815 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.740928 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.741076 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.741549 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.741647 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":89,"threads_started":1}
I20260929 10:27:18.741851 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.735247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:37551 (local address 127.0.0.1:37316)
0929 10:27:18.736148 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.736199 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.736328 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.737002 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.737004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.737024 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:18.737237 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.737245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.738844 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.738847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.740130 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.740141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.740979 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.740981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.741238 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.741496 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.741559 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.741637 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":292,"spinlock_wait_cycles":14208,"thread_start_us":81,"threads_started":1}
I20260929 10:27:18.742193 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.735542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:46823 (local address 127.0.0.1:58770)
0929 10:27:18.736148 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.736200 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.736329 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.736986 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.736993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.737020 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:18.737237 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.737245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.738470 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.738476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.739890 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.739899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.740951 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.740958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.741243 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.741307 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.741416 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.742053 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":669,"spinlock_wait_cycles":11520,"thread_start_us":111,"threads_started":1}
I20260929 10:27:18.741712 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.735247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:34539 (local address 127.0.0.1:49920)
0929 10:27:18.736733 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.736745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.736836 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.737515 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.737519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.737536 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:18.737639 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.737645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.739042 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.739045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.740170 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.740181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.741176 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.741180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.741291 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.741313 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.741425 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.741558 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1430,"mutex_wait_us":500,"thread_start_us":93,"threads_started":1}
I20260929 10:27:18.742403 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.735486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.21.33.130:37551)
0929 10:27:18.736533 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.736548 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.736569 (+ 21us) server_negotiation.cc:407] Connection header received
0929 10:27:18.736647 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.736655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.736861 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.737399 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:18.737549 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.738682 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.740283 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.740834 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.741313 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.741463 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.742139 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.742248 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":114,"threads_started":1}
I20260929 10:27:18.744884 7430 tablet_service.cc:1467] Processing CreateTablet for tablet 4d09f251f6074eb6bf8d57d1495e3c01 (DEFAULT_TABLE table=test-table [id=decce0c13a694365b189a281491a1b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:18.745044 7698 tablet_service.cc:1467] Processing CreateTablet for tablet 4d09f251f6074eb6bf8d57d1495e3c01 (DEFAULT_TABLE table=test-table [id=decce0c13a694365b189a281491a1b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:18.745044 7545 tablet_service.cc:1467] Processing CreateTablet for tablet 4d09f251f6074eb6bf8d57d1495e3c01 (DEFAULT_TABLE table=test-table [id=decce0c13a694365b189a281491a1b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:18.746086 7698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d09f251f6074eb6bf8d57d1495e3c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:18.746088 7430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d09f251f6074eb6bf8d57d1495e3c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:18.746086 7545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d09f251f6074eb6bf8d57d1495e3c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:18.752350 7781 tablet_bootstrap.cc:492] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940: Bootstrap starting.
I20260929 10:27:18.753921 7783 tablet_bootstrap.cc:492] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Bootstrap starting.
I20260929 10:27:18.753921 7782 tablet_bootstrap.cc:492] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Bootstrap starting.
I20260929 10:27:18.754683 7781 tablet_bootstrap.cc:654] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:18.755404 7781 log.cc:824] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:18.756075 7782 tablet_bootstrap.cc:654] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:18.756075 7783 tablet_bootstrap.cc:654] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:18.757460 7783 log.cc:824] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:18.757460 7782 log.cc:824] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:18.757867 7781 tablet_bootstrap.cc:492] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940: No bootstrap required, opened a new log
I20260929 10:27:18.758069 7781 ts_tablet_manager.cc:1397] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260929 10:27:18.759171 7783 tablet_bootstrap.cc:492] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: No bootstrap required, opened a new log
I20260929 10:27:18.759373 7783 ts_tablet_manager.cc:1397] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:27:18.759423 7782 tablet_bootstrap.cc:492] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: No bootstrap required, opened a new log
I20260929 10:27:18.759629 7782 ts_tablet_manager.cc:1397] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:27:18.762212 7783 raft_consensus.cc:348] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.762454 7783 raft_consensus.cc:374] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:18.762513 7783 raft_consensus.cc:729] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4934b66acf4a8ab4e12a7bd9feef04, State: Initialized, Role: FOLLOWER
I20260929 10:27:18.762530 7781 raft_consensus.cc:348] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.762800 7781 raft_consensus.cc:374] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:18.762856 7781 raft_consensus.cc:729] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c500a1d4983b4a6a97a009fadf039940, State: Initialized, Role: FOLLOWER
I20260929 10:27:18.763023 7783 consensus_queue.cc:260] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.763326 7781 consensus_queue.cc:260] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.763743 7764 heartbeater.cc:503] Master 127.21.33.190:42407 was elected leader, sending a full tablet report...
I20260929 10:27:18.764079 7783 ts_tablet_manager.cc:1428] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:27:18.763943 7782 raft_consensus.cc:348] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.764179 7782 raft_consensus.cc:374] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:18.764240 7782 raft_consensus.cc:729] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6f546d9ad04c34b24f28b77b312f29, State: Initialized, Role: FOLLOWER
I20260929 10:27:18.764696 7781 ts_tablet_manager.cc:1428] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 10:27:18.764649 7782 consensus_queue.cc:260] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.765023 7496 heartbeater.cc:503] Master 127.21.33.190:42407 was elected leader, sending a full tablet report...
I20260929 10:27:18.765551 7782 ts_tablet_manager.cc:1428] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 10:27:18.766188 7630 heartbeater.cc:503] Master 127.21.33.190:42407 was elected leader, sending a full tablet report...
I20260929 10:27:18.832001 7789 raft_consensus.cc:482] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:18.832203 7789 raft_consensus.cc:504] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.833318 7789 leader_election.cc:288] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c4934b66acf4a8ab4e12a7bd9feef04 (127.21.33.131:34539), c500a1d4983b4a6a97a009fadf039940 (127.21.33.129:46823)
I20260929 10:27:18.838449 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.833649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:59735 (local address 127.21.33.129:46823)
0929 10:27:18.833773 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.833778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.833796 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:27:18.833929 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.833932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.834001 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.834089 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:18.834426 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.835246 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.837029 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.837689 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.837813 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.837864 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.838247 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.838322 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:27:18.838449 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.833652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:41077 (local address 127.21.33.131:34539)
0929 10:27:18.833774 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.833779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.834003 (+ 224us) server_negotiation.cc:407] Connection header received
0929 10:27:18.834063 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.834066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.834117 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.834190 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:18.834450 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.835240 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.837029 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.837664 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.837813 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.837864 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.838247 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.838320 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:27:18.838636 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.833485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:46823 (local address 127.21.33.130:59735)
0929 10:27:18.833708 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.833724 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.833816 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.834122 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.834125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.834138 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:18.834278 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.834285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.835377 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.835381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.836881 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.836893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.837826 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.837831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.838064 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.838124 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.838220 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.838386 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":180}
I20260929 10:27:18.838930 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.833485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:34539 (local address 127.21.33.130:41077)
0929 10:27:18.833904 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.833918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:18.834000 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:18.834211 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:18.834213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:18.834225 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:18.834336 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.834344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.835377 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.835381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.836881 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:18.836893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.837851 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.837854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:18.838069 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.838143 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 10:27:18.838220 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:27:18.838386 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":305,"thread_start_us":131,"threads_started":1}
I20260929 10:27:18.839475 7450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d09f251f6074eb6bf8d57d1495e3c01" candidate_uuid: "ab6f546d9ad04c34b24f28b77b312f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c500a1d4983b4a6a97a009fadf039940" is_pre_election: true
I20260929 10:27:18.839533 7718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d09f251f6074eb6bf8d57d1495e3c01" candidate_uuid: "ab6f546d9ad04c34b24f28b77b312f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" is_pre_election: true
I20260929 10:27:18.839819 7718 raft_consensus.cc:2454] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6f546d9ad04c34b24f28b77b312f29 in term 0.
I20260929 10:27:18.839820 7450 raft_consensus.cc:2454] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6f546d9ad04c34b24f28b77b312f29 in term 0.
I20260929 10:27:18.840350 7519 leader_election.cc:302] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8c4934b66acf4a8ab4e12a7bd9feef04, ab6f546d9ad04c34b24f28b77b312f29, c500a1d4983b4a6a97a009fadf039940; no voters:
I20260929 10:27:18.840668 7789 raft_consensus.cc:2781] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:27:18.840765 7789 raft_consensus.cc:482] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:27:18.840827 7789 raft_consensus.cc:3037] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:18.841864 7789 raft_consensus.cc:504] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.842221 7789 leader_election.cc:288] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [CANDIDATE]: Term 1 election: Requested vote from peers 8c4934b66acf4a8ab4e12a7bd9feef04 (127.21.33.131:34539), c500a1d4983b4a6a97a009fadf039940 (127.21.33.129:46823)
I20260929 10:27:18.842630 7450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d09f251f6074eb6bf8d57d1495e3c01" candidate_uuid: "ab6f546d9ad04c34b24f28b77b312f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c500a1d4983b4a6a97a009fadf039940"
I20260929 10:27:18.842758 7718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d09f251f6074eb6bf8d57d1495e3c01" candidate_uuid: "ab6f546d9ad04c34b24f28b77b312f29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04"
I20260929 10:27:18.842746 7450 raft_consensus.cc:3037] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:18.842900 7718 raft_consensus.cc:3037] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:18.843915 7718 raft_consensus.cc:2454] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6f546d9ad04c34b24f28b77b312f29 in term 1.
I20260929 10:27:18.843993 7450 raft_consensus.cc:2454] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6f546d9ad04c34b24f28b77b312f29 in term 1.
I20260929 10:27:18.844286 7520 leader_election.cc:302] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c4934b66acf4a8ab4e12a7bd9feef04, ab6f546d9ad04c34b24f28b77b312f29; no voters:
I20260929 10:27:18.844491 7789 raft_consensus.cc:2781] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:18.844728 7789 raft_consensus.cc:686] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 1 LEADER]: Becoming Leader. State: Replica: ab6f546d9ad04c34b24f28b77b312f29, State: Running, Role: LEADER
I20260929 10:27:18.845060 7789 consensus_queue.cc:237] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } }
I20260929 10:27:18.848258 7304 catalog_manager.cc:5495] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6f546d9ad04c34b24f28b77b312f29 (127.21.33.130). New cstate: current_term: 1 leader_uuid: "ab6f546d9ad04c34b24f28b77b312f29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6f546d9ad04c34b24f28b77b312f29" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 37551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 } health_report { overall_health: UNKNOWN } } }
I20260929 10:27:18.863423 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.860076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.21.33.130:37551)
0929 10:27:18.860200 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:18.860203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:18.860216 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:27:18.860307 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:18.860309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:18.860362 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:18.860450 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:18.860672 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.861187 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.862037 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:18.862271 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:18.862348 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:18.862610 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:18.863008 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:18.863125 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:18.863254 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 10:27:18.863289 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:27:18.869623 7718 raft_consensus.cc:1261] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 1 FOLLOWER]: Refusing update from remote peer ab6f546d9ad04c34b24f28b77b312f29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:18.869632 7450 raft_consensus.cc:1261] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 1 FOLLOWER]: Refusing update from remote peer ab6f546d9ad04c34b24f28b77b312f29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:18.870373 7789 consensus_queue.cc:1035] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c500a1d4983b4a6a97a009fadf039940" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:18.870580 7791 consensus_queue.cc:1035] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c4934b66acf4a8ab4e12a7bd9feef04" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:18.879061 7796 mvcc.cc:205] Tried to move back new op lower bound from 7334615608801890304 to 7334615608711909376. Current Snapshot: MvccSnapshot[applied={T|T < 7334615608801890304}]
I20260929 10:27:18.879315 7795 mvcc.cc:205] Tried to move back new op lower bound from 7334615608801890304 to 7334615608711909376. Current Snapshot: MvccSnapshot[applied={T|T < 7334615608801890304}]
I20260929 10:27:18.889029 7304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:18.889262 7304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:18.891916 7304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bde7f40a1d3a44eabeb6de5ed7cb24dd: Sending DeleteTablet for 3 replicas of tablet 4d09f251f6074eb6bf8d57d1495e3c01
I20260929 10:27:18.892725 7698 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d09f251f6074eb6bf8d57d1495e3c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:18 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:49920
I20260929 10:27:18.892719 7545 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d09f251f6074eb6bf8d57d1495e3c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:18 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:37316
I20260929 10:27:18.892942 7430 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d09f251f6074eb6bf8d57d1495e3c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:18 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:58770
I20260929 10:27:18.893330 7804 tablet_replica.cc:330] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: stopping tablet replica
I20260929 10:27:18.893467 7805 tablet_replica.cc:330] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: stopping tablet replica
I20260929 10:27:18.893769 7805 raft_consensus.cc:2229] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:27:18.893872 7806 tablet_replica.cc:330] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940: stopping tablet replica
I20260929 10:27:18.894143 7806 raft_consensus.cc:2229] T 4d09f251f6074eb6bf8d57d1495e3c01 P c500a1d4983b4a6a97a009fadf039940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:18.894194 7805 raft_consensus.cc:2258] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:18.894308 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 7367
I20260929 10:27:18.896929 7805 ts_tablet_manager.cc:1905] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:18.897369 7804 raft_consensus.cc:2229] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:18.897719 7804 raft_consensus.cc:2258] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:18.899307 7804 ts_tablet_manager.cc:1905] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:18.899931 7805 ts_tablet_manager.cc:1918] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:18.900035 7805 log.cc:1196] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/4d09f251f6074eb6bf8d57d1495e3c01
I20260929 10:27:18.900377 7805 ts_tablet_manager.cc:1939] T 4d09f251f6074eb6bf8d57d1495e3c01 P ab6f546d9ad04c34b24f28b77b312f29: Deleting consensus metadata
I20260929 10:27:18.901675 7290 catalog_manager.cc:4841] TS ab6f546d9ad04c34b24f28b77b312f29 (127.21.33.130:37551): tablet 4d09f251f6074eb6bf8d57d1495e3c01 (table test-table [id=decce0c13a694365b189a281491a1b9c]) successfully deleted
I20260929 10:27:18.902740 7804 ts_tablet_manager.cc:1918] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:18.902848 7804 log.cc:1196] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/4d09f251f6074eb6bf8d57d1495e3c01
I20260929 10:27:18.903193 7804 ts_tablet_manager.cc:1939] T 4d09f251f6074eb6bf8d57d1495e3c01 P 8c4934b66acf4a8ab4e12a7bd9feef04: Deleting consensus metadata
W20260929 10:27:18.903465 7291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:46823 (error 108)
I20260929 10:27:18.904295 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 7501
I20260929 10:27:18.904458 7292 catalog_manager.cc:4841] TS 8c4934b66acf4a8ab4e12a7bd9feef04 (127.21.33.131:34539): tablet 4d09f251f6074eb6bf8d57d1495e3c01 (table test-table [id=decce0c13a694365b189a281491a1b9c]) successfully deleted
I20260929 10:27:18.905078 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:18.904213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:46823 (local address 127.0.0.1:58780)
0929 10:27:18.904885 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:18.904965 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:46823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572}
W20260929 10:27:18.905300 7291 catalog_manager.cc:4584] TS c500a1d4983b4a6a97a009fadf039940 (127.21.33.129:46823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d09f251f6074eb6bf8d57d1495e3c01: Network error: Client connection negotiation failed: client connection to 127.21.33.129:46823: connect: Connection refused (error 111)
I20260929 10:27:18.910334 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 7635
I20260929 10:27:18.915799 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 7273
2026-09-29T10:27:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2306 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T10:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:18Z Disabled control of system clock
I20260929 10:27:18.946175 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:44581
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:33937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:27:19.058054 7811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:19.058331 7811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:19.058367 7811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:19.062386 7811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:19.062443 7811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:19.062461 7811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:19.062486 7811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:19.062502 7811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:19.067322 7811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:33937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:44581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:19.068426 7811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:19.069847 7811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:19.078792 7816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:19.078761 7819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:19.078840 7817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:19.078971 7811 server_base.cc:1034] running on GCE node
I20260929 10:27:19.079955 7811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:19.081089 7811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:19.082254 7811 hybrid_clock.cc:648] HybridClock initialized: now 1790677639082241 us; error 37 us; skew 500 ppm
I20260929 10:27:19.084154 7811 webserver.cc:460] Webserver started at http://127.21.33.190:42229/ using document root <none> and password file <none>
I20260929 10:27:19.084750 7811 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:19.084813 7811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:19.085052 7811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:19.087154 7811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "5a802fda66bc482eb472af26f65df804"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "f32447716b0a54cb8ca079c7378b817a"
server_key_iv: "b8224a4bf866f64b42b9923d4ade2b7f"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.087736 7811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "5a802fda66bc482eb472af26f65df804"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "f32447716b0a54cb8ca079c7378b817a"
server_key_iv: "b8224a4bf866f64b42b9923d4ade2b7f"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.091512 7811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:27:19.093924 7825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.095150 7811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:19.095285 7811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "5a802fda66bc482eb472af26f65df804"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "f32447716b0a54cb8ca079c7378b817a"
server_key_iv: "b8224a4bf866f64b42b9923d4ade2b7f"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.095395 7811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:19.107893 7811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:19.108548 7811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:19.108747 7811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:19.115507 7811 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:44581
I20260929 10:27:19.115516 7877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:44581 every 8 connection(s)
I20260929 10:27:19.116420 7811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:19.119441 7878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:19.122085 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 7811
I20260929 10:27:19.122216 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:19.122493 21638 external_mini_cluster.cc:1428] Setting key d90e6d5b41207ee1a68a53ed1da1ab50
I20260929 10:27:19.125506 7878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804: Bootstrap starting.
I20260929 10:27:19.127658 7878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:19.128386 7878 log.cc:824] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:19.130321 7878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804: No bootstrap required, opened a new log
I20260929 10:27:19.131851 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.123911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.21.33.190:44581)
0929 10:27:19.124556 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.124574 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.124608 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:27:19.125265 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.125283 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.125723 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.126057 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:19.126497 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.127450 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.128408 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.129076 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.129278 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.130130 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:19.130136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:19.130162 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:19.130608 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:19.130664 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:19.130685 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:19.130792 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.130909 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.131173 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260929 10:27:19.134220 7878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a802fda66bc482eb472af26f65df804" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44581 } }
I20260929 10:27:19.134485 7878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:19.134559 7878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a802fda66bc482eb472af26f65df804, State: Initialized, Role: FOLLOWER
I20260929 10:27:19.134958 7878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a802fda66bc482eb472af26f65df804" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44581 } }
I20260929 10:27:19.135099 7878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:19.135188 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:19.135286 7878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:19.136181 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a802fda66bc482eb472af26f65df804" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44581 } }
I20260929 10:27:19.136520 7878 leader_election.cc:302] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a802fda66bc482eb472af26f65df804; no voters:
I20260929 10:27:19.136826 7878 leader_election.cc:288] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:19.136942 7883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:19.137238 7883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [term 1 LEADER]: Becoming Leader. State: Replica: 5a802fda66bc482eb472af26f65df804, State: Running, Role: LEADER
I20260929 10:27:19.137569 7883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a802fda66bc482eb472af26f65df804" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44581 } }
I20260929 10:27:19.138098 7878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:19.139212 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a802fda66bc482eb472af26f65df804. Latest consensus state: current_term: 1 leader_uuid: "5a802fda66bc482eb472af26f65df804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a802fda66bc482eb472af26f65df804" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44581 } } }
I20260929 10:27:19.139340 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:19.139618 7884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a802fda66bc482eb472af26f65df804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a802fda66bc482eb472af26f65df804" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 44581 } } }
I20260929 10:27:19.139725 7884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:19.139823 7892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:19.143121 7892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:19.148577 7892 catalog_manager.cc:1347] Generated new cluster ID: e06fb49fafee4784913dbda6fada4068
I20260929 10:27:19.148664 7892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:19.169456 7892 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:19.171468 7892 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:19.183563 7892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804: Generated new TSK 0
I20260929 10:27:19.184554 7892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:19.188812 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44581
--builtin_ntp_servers=127.21.33.148:33937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:27:19.304155 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:19.304418 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:19.304456 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:19.308480 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:19.308533 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:19.308588 7902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:27:19.313139 7902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:33937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:19.314126 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:19.315526 7902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:19.325099 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:19.325135 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:19.325286 7902 server_base.cc:1034] running on GCE node
W20260929 10:27:19.325099 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:19.325816 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:19.326643 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:19.327836 7902 hybrid_clock.cc:648] HybridClock initialized: now 1790677639327820 us; error 28 us; skew 500 ppm
I20260929 10:27:19.329664 7902 webserver.cc:460] Webserver started at http://127.21.33.129:45171/ using document root <none> and password file <none>
I20260929 10:27:19.330247 7902 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:19.330336 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:19.330564 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:19.332543 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "51b7c73f3fb549dda42748b879cc410f"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "5595a1f0a8c9b85a69bb94c6f2b51ebc"
server_key_iv: "ae50ff1bdb1d71ce6ff4aaf7636e712a"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.333097 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "51b7c73f3fb549dda42748b879cc410f"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "5595a1f0a8c9b85a69bb94c6f2b51ebc"
server_key_iv: "ae50ff1bdb1d71ce6ff4aaf7636e712a"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.336630 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:27:19.339103 7916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.340325 7902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:19.340564 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "51b7c73f3fb549dda42748b879cc410f"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "5595a1f0a8c9b85a69bb94c6f2b51ebc"
server_key_iv: "ae50ff1bdb1d71ce6ff4aaf7636e712a"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.340683 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:19.353037 7902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:19.353799 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:19.354003 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:19.354585 7902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:19.355553 7902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:19.355626 7902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.355671 7902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:19.355700 7902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.365315 7902 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:34983
I20260929 10:27:19.365343 8029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:34983 every 8 connection(s)
I20260929 10:27:19.366290 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:27:19.375348 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.368072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:55323 (local address 127.21.33.190:44581)
0929 10:27:19.368261 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.368264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.369305 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 10:27:19.370711 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.370714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.370771 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.370854 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:19.371718 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.372258 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.373473 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.373725 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.373817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.374942 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:19.374945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:19.374947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:19.374959 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:19.374975 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:19.374982 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:19.375040 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.375179 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.375229 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 10:27:19.376055 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 7902
I20260929 10:27:19.376207 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:27:19.376327 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.368476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:44581 (local address 127.21.33.129:55323)
0929 10:27:19.369136 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.369222 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.370489 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.371004 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.371011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.371095 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:19.371594 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.371605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.372367 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.372369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.373359 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.373369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.373864 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.373866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.374006 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.374697 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:19.374699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:19.374740 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:19.374804 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:19.375057 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:19.375063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:19.375067 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.375178 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.375420 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20260929 10:27:19.376569 21638 external_mini_cluster.cc:1428] Setting key 7fbf8bda82e392704391beecd89f3496
I20260929 10:27:19.377884 8030 heartbeater.cc:347] Connected to a master server at 127.21.33.190:44581
I20260929 10:27:19.378159 8030 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:19.378676 8030 heartbeater.cc:511] Master 127.21.33.190:44581 requested a full tablet report, sending...
I20260929 10:27:19.378679 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44581
--builtin_ntp_servers=127.21.33.148:33937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:27:19.380184 7842 ts_manager.cc:194] Registered new tserver with Master: 51b7c73f3fb549dda42748b879cc410f (127.21.33.129:34983)
I20260929 10:27:19.381399 7842 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.129:55323
W20260929 10:27:19.491240 8033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:19.491493 8033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:19.491530 8033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:19.495569 8033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:19.495625 8033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:19.495685 8033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:27:19.500576 8033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:33937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:19.501703 8033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:19.503099 8033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:19.512639 8039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:19.512650 8038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:19.512670 8041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:19.512696 8033 server_base.cc:1034] running on GCE node
I20260929 10:27:19.513298 8033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:19.513865 8033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:19.515022 8033 hybrid_clock.cc:648] HybridClock initialized: now 1790677639514987 us; error 46 us; skew 500 ppm
I20260929 10:27:19.516834 8033 webserver.cc:460] Webserver started at http://127.21.33.130:44675/ using document root <none> and password file <none>
I20260929 10:27:19.517472 8033 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:19.517535 8033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:19.517771 8033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:19.519769 8033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "ac73989afb9041aab439550a6df35e0e"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "be85dd58043f4b76616f911768d6b44f"
server_key_iv: "5694ecae842fcbcf6af3b9b65ed18230"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.520339 8033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "ac73989afb9041aab439550a6df35e0e"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "be85dd58043f4b76616f911768d6b44f"
server_key_iv: "5694ecae842fcbcf6af3b9b65ed18230"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.523862 8033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:27:19.526291 8047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.527465 8033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:27:19.527592 8033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "ac73989afb9041aab439550a6df35e0e"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "be85dd58043f4b76616f911768d6b44f"
server_key_iv: "5694ecae842fcbcf6af3b9b65ed18230"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.527699 8033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:19.566722 8033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:19.567443 8033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:19.567656 8033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:19.568264 8033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:19.569247 8033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:19.569315 8033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.569386 8033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:19.569434 8033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.578333 8033 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:43005
I20260929 10:27:19.578370 8160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:43005 every 8 connection(s)
I20260929 10:27:19.579242 8033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:27:19.585726 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8033
I20260929 10:27:19.585876 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:27:19.586159 21638 external_mini_cluster.cc:1428] Setting key 94aff7722e15615c4b45bb3d42fc9e65
I20260929 10:27:19.588358 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44581
--builtin_ntp_servers=127.21.33.148:33937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:27:19.588578 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.581095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:48217 (local address 127.21.33.190:44581)
0929 10:27:19.581300 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.581304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.582266 (+ 962us) server_negotiation.cc:407] Connection header received
0929 10:27:19.583742 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.583745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.583792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.583870 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:19.584705 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.585299 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.586529 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.586759 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.586874 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.588041 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:19.588045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:19.588047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:19.588058 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:19.588078 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:19.588085 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:19.588141 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.588327 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.588379 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 10:27:19.589725 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.581507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:44581 (local address 127.21.33.130:48217)
0929 10:27:19.582146 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.582185 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.583523 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.584029 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.584035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.584114 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:19.584572 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.584584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.585407 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.585409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.586409 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.586419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.586914 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.586918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.587078 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.587752 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:19.587754 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:19.587807 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:19.587883 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:19.588169 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:19.588180 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:19.588190 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.588325 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.588637 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260929 10:27:19.590871 8161 heartbeater.cc:347] Connected to a master server at 127.21.33.190:44581
I20260929 10:27:19.591121 8161 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:19.591720 8161 heartbeater.cc:511] Master 127.21.33.190:44581 requested a full tablet report, sending...
I20260929 10:27:19.592814 7842 ts_manager.cc:194] Registered new tserver with Master: ac73989afb9041aab439550a6df35e0e (127.21.33.130:43005)
I20260929 10:27:19.593540 7842 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.130:48217
W20260929 10:27:19.700737 8164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:19.701007 8164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:19.701045 8164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:19.705073 8164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:19.705137 8164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:19.705241 8164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:27:19.710213 8164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:33937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:44581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:19.711305 8164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:19.712692 8164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:19.722335 8172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:19.722370 8170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:19.722334 8169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:19.722611 8164 server_base.cc:1034] running on GCE node
I20260929 10:27:19.723097 8164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:19.723727 8164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:19.724887 8164 hybrid_clock.cc:648] HybridClock initialized: now 1790677639724870 us; error 27 us; skew 500 ppm
I20260929 10:27:19.726848 8164 webserver.cc:460] Webserver started at http://127.21.33.131:37985/ using document root <none> and password file <none>
I20260929 10:27:19.727414 8164 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:19.727475 8164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:19.727742 8164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:19.729843 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "9e391f2290724eb2bb4b249070962f2a"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "adadf73760f925f5aa2e1cdcf50d1177"
server_key_iv: "e093cf640159066f725cc277e26c4aea"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.730392 8164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "9e391f2290724eb2bb4b249070962f2a"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "adadf73760f925f5aa2e1cdcf50d1177"
server_key_iv: "e093cf640159066f725cc277e26c4aea"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.734016 8164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:27:19.736280 8178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.737527 8164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:19.737660 8164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "9e391f2290724eb2bb4b249070962f2a"
format_stamp: "Formatted at 2026-09-29 10:27:19 on dist-test-slave-2rbl"
server_key: "adadf73760f925f5aa2e1cdcf50d1177"
server_key_iv: "e093cf640159066f725cc277e26c4aea"
server_key_version: "encryptionkey@0"
I20260929 10:27:19.737771 8164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:19.749682 8164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:19.750320 8164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:19.750517 8164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:19.751098 8164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:19.752007 8164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:19.752075 8164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.752141 8164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:19.752197 8164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:19.761359 8164 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:33739
I20260929 10:27:19.761423 8291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:33739 every 8 connection(s)
I20260929 10:27:19.762334 8164 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:27:19.765481 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8164
I20260929 10:27:19.765617 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:27:19.765892 21638 external_mini_cluster.cc:1428] Setting key 8787dd1d4ad30fdf800436f6df273b5d
I20260929 10:27:19.772327 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.764169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:45897 (local address 127.21.33.190:44581)
0929 10:27:19.764303 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.764307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.765450 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 10:27:19.767221 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.767225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.767279 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.767380 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:19.768475 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.769003 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.770333 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.770577 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.770678 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.771885 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:19.771888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:19.771891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:19.771903 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:19.771924 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:19.771932 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:19.772005 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.772155 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.772211 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:27:19.773270 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.764603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:44581 (local address 127.21.33.131:45897)
0929 10:27:19.765296 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.765338 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.766967 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.767570 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.767580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.767699 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:19.768325 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.768340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.769161 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.769164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.770193 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.770202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.770735 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.770739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.770895 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.771619 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:19.771621 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:19.771662 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:19.771725 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:19.772005 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:19.772010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:19.772013 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.772134 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.772354 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20260929 10:27:19.774473 8292 heartbeater.cc:347] Connected to a master server at 127.21.33.190:44581
I20260929 10:27:19.774732 8292 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:19.775250 8292 heartbeater.cc:511] Master 127.21.33.190:44581 requested a full tablet report, sending...
I20260929 10:27:19.776280 7842 ts_manager.cc:194] Registered new tserver with Master: 9e391f2290724eb2bb4b249070962f2a (127.21.33.131:33739)
I20260929 10:27:19.776880 7842 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.131:45897
I20260929 10:27:19.780076 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:27:19.786307 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.782113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.21.33.190:44581)
0929 10:27:19.782277 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.782281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.782496 (+ 215us) server_negotiation.cc:407] Connection header received
0929 10:27:19.782646 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.782651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.782703 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.782794 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:19.783084 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.783675 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.784545 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.784775 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.784885 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.785749 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:19.785752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:19.785755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:19.785768 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:19.785794 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:19.785802 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:19.785935 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.786114 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.786154 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 10:27:19.789352 7842 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:27:19.792327 7842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:27:19.807463 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.801123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:43005 (local address 127.0.0.1:58832)
0929 10:27:19.802539 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.802554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.802634 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.803048 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.803051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.803077 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:19.803214 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.803221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.804624 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.804627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.806044 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.806053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.806870 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.806874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.807019 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.807057 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.807174 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.807295 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":107,"spinlock_wait_cycles":768,"thread_start_us":110,"threads_started":1}
I20260929 10:27:19.807466 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.801327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.21.33.130:43005)
0929 10:27:19.801719 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.801731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.802658 (+ 927us) server_negotiation.cc:407] Connection header received
0929 10:27:19.802745 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.802751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.802910 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.803061 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:19.803361 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.804504 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.806199 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.806748 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.806871 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.807080 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.807205 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.807316 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":154,"threads_started":1}
I20260929 10:27:19.808190 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.802045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.21.33.131:33739)
0929 10:27:19.803354 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.803365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.803394 (+ 29us) server_negotiation.cc:407] Connection header received
0929 10:27:19.803451 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.803454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.803575 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.803721 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:19.803928 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.804699 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.806221 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.806996 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.807110 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.807290 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.807937 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.808035 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":85,"threads_started":1}
I20260929 10:27:19.808717 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.801887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:33739 (local address 127.0.0.1:44986)
0929 10:27:19.802313 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.802326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.802433 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.803699 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.803702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.803719 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:19.803797 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.803803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.804828 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.804831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.806083 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.806093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.807355 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.807358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.807445 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.807466 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.808492 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.808569 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":7,"spinlock_wait_cycles":1920,"thread_start_us":95,"threads_started":1}
I20260929 10:27:19.809273 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.801775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:34983 (local address 127.0.0.1:43876)
0929 10:27:19.802244 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.802295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.802431 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.803001 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.803006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.803027 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:19.803197 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.803204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.805837 (+ 2633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.805841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.806981 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.806992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.808276 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.808281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.808954 (+ 673us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.808978 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.809057 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.809122 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":7,"thread_start_us":85,"threads_started":1}
I20260929 10:27:19.809405 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.801471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.21.33.129:34983)
0929 10:27:19.801940 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.801953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.802387 (+ 434us) server_negotiation.cc:407] Connection header received
0929 10:27:19.802668 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.802674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.802852 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.803012 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:19.803307 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.804504 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.807124 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.807913 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.807992 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.808164 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.809132 (+ 968us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.809254 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":89,"threads_started":1}
I20260929 10:27:19.810727 8095 tablet_service.cc:1467] Processing CreateTablet for tablet feec82a5387240da8b55eab12cf7560a (DEFAULT_TABLE table=test-table [id=db0c603d532a4db39575a216588831b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:19.811743 8095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feec82a5387240da8b55eab12cf7560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:19.811404 8226 tablet_service.cc:1467] Processing CreateTablet for tablet feec82a5387240da8b55eab12cf7560a (DEFAULT_TABLE table=test-table [id=db0c603d532a4db39575a216588831b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:19.811924 7964 tablet_service.cc:1467] Processing CreateTablet for tablet feec82a5387240da8b55eab12cf7560a (DEFAULT_TABLE table=test-table [id=db0c603d532a4db39575a216588831b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:19.812419 8226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feec82a5387240da8b55eab12cf7560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:19.812875 7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feec82a5387240da8b55eab12cf7560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:19.818076 8309 tablet_bootstrap.cc:492] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Bootstrap starting.
I20260929 10:27:19.819816 8310 tablet_bootstrap.cc:492] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: Bootstrap starting.
I20260929 10:27:19.820345 8309 tablet_bootstrap.cc:654] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:19.821059 8309 log.cc:824] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:19.821555 8311 tablet_bootstrap.cc:492] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Bootstrap starting.
I20260929 10:27:19.821719 8310 tablet_bootstrap.cc:654] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:19.822399 8310 log.cc:824] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:19.823320 8309 tablet_bootstrap.cc:492] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: No bootstrap required, opened a new log
I20260929 10:27:19.823519 8309 ts_tablet_manager.cc:1397] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:27:19.823616 8311 tablet_bootstrap.cc:654] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:19.824424 8310 tablet_bootstrap.cc:492] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: No bootstrap required, opened a new log
I20260929 10:27:19.824450 8311 log.cc:824] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:19.824631 8310 ts_tablet_manager.cc:1397] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:27:19.826421 8311 tablet_bootstrap.cc:492] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: No bootstrap required, opened a new log
I20260929 10:27:19.826602 8311 ts_tablet_manager.cc:1397] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:27:19.827828 8309 raft_consensus.cc:348] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.828104 8309 raft_consensus.cc:374] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:19.827947 8310 raft_consensus.cc:348] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.828158 8309 raft_consensus.cc:729] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac73989afb9041aab439550a6df35e0e, State: Initialized, Role: FOLLOWER
I20260929 10:27:19.828173 8310 raft_consensus.cc:374] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:19.828226 8310 raft_consensus.cc:729] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b7c73f3fb549dda42748b879cc410f, State: Initialized, Role: FOLLOWER
I20260929 10:27:19.828640 8310 consensus_queue.cc:260] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.828605 8309 consensus_queue.cc:260] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.829579 8161 heartbeater.cc:503] Master 127.21.33.190:44581 was elected leader, sending a full tablet report...
I20260929 10:27:19.829677 8310 ts_tablet_manager.cc:1428] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 10:27:19.829679 8309 ts_tablet_manager.cc:1428] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 10:27:19.830102 8162 tablet.cc:2369] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:27:19.830811 8311 raft_consensus.cc:348] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.831043 8030 heartbeater.cc:503] Master 127.21.33.190:44581 was elected leader, sending a full tablet report...
I20260929 10:27:19.831081 8311 raft_consensus.cc:374] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:19.831173 8311 raft_consensus.cc:729] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e391f2290724eb2bb4b249070962f2a, State: Initialized, Role: FOLLOWER
I20260929 10:27:19.831599 8311 consensus_queue.cc:260] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.832280 8292 heartbeater.cc:503] Master 127.21.33.190:44581 was elected leader, sending a full tablet report...
I20260929 10:27:19.832554 8311 ts_tablet_manager.cc:1428] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 10:27:19.867408 8031 tablet.cc:2369] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:27:19.883319 8316 raft_consensus.cc:482] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:19.883630 8316 raft_consensus.cc:504] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.884794 8316 leader_election.cc:288] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51b7c73f3fb549dda42748b879cc410f (127.21.33.129:34983), 9e391f2290724eb2bb4b249070962f2a (127.21.33.131:33739)
I20260929 10:27:19.888720 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.885092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:33739 (local address 127.21.33.130:40283)
0929 10:27:19.885312 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.885329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.885423 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.885756 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.885759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.885782 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:19.885911 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.885918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.886750 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.886753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.887572 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.887581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.888296 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.888301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.888401 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.888424 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.888496 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.888551 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 10:27:19.888737 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.885236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:40283 (local address 127.21.33.131:33739)
0929 10:27:19.885369 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.885375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.885394 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:27:19.885553 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.885556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.885622 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.885722 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:19.886023 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.886636 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.887711 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.888169 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.888261 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.888298 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.888535 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.888616 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:27:19.888826 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.885240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:48391 (local address 127.21.33.129:34983)
0929 10:27:19.885378 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.885383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.885730 (+ 347us) server_negotiation.cc:407] Connection header received
0929 10:27:19.885859 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.885862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.885911 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.885994 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:19.886280 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.886796 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.887832 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.888251 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.888365 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.888415 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.888632 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.888700 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 10:27:19.888988 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.885133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:34983 (local address 127.21.33.130:48391)
0929 10:27:19.885622 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.885636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:19.885753 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:19.886022 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:19.886025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:19.886041 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:19.886165 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.886171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.886904 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.886906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.887715 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:19.887725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.888387 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.888390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:19.888467 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.888512 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 10:27:19.888592 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 10:27:19.888652 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260929 10:27:19.889483 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec82a5387240da8b55eab12cf7560a" candidate_uuid: "ac73989afb9041aab439550a6df35e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e391f2290724eb2bb4b249070962f2a" is_pre_election: true
I20260929 10:27:19.889544 7984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec82a5387240da8b55eab12cf7560a" candidate_uuid: "ac73989afb9041aab439550a6df35e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b7c73f3fb549dda42748b879cc410f" is_pre_election: true
I20260929 10:27:19.889803 8246 raft_consensus.cc:2454] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac73989afb9041aab439550a6df35e0e in term 0.
I20260929 10:27:19.889808 7984 raft_consensus.cc:2454] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac73989afb9041aab439550a6df35e0e in term 0.
I20260929 10:27:19.890342 8051 leader_election.cc:302] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51b7c73f3fb549dda42748b879cc410f, 9e391f2290724eb2bb4b249070962f2a, ac73989afb9041aab439550a6df35e0e; no voters:
I20260929 10:27:19.890599 8316 raft_consensus.cc:2781] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:27:19.890682 8316 raft_consensus.cc:482] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:27:19.890726 8316 raft_consensus.cc:3037] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:19.891757 8316 raft_consensus.cc:504] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.892114 8316 leader_election.cc:288] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [CANDIDATE]: Term 1 election: Requested vote from peers 51b7c73f3fb549dda42748b879cc410f (127.21.33.129:34983), 9e391f2290724eb2bb4b249070962f2a (127.21.33.131:33739)
I20260929 10:27:19.892400 7984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec82a5387240da8b55eab12cf7560a" candidate_uuid: "ac73989afb9041aab439550a6df35e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51b7c73f3fb549dda42748b879cc410f"
I20260929 10:27:19.892551 7984 raft_consensus.cc:3037] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:19.892565 8246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feec82a5387240da8b55eab12cf7560a" candidate_uuid: "ac73989afb9041aab439550a6df35e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e391f2290724eb2bb4b249070962f2a"
I20260929 10:27:19.892699 8246 raft_consensus.cc:3037] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:19.893702 7984 raft_consensus.cc:2454] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac73989afb9041aab439550a6df35e0e in term 1.
I20260929 10:27:19.893893 8246 raft_consensus.cc:2454] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac73989afb9041aab439550a6df35e0e in term 1.
I20260929 10:27:19.894040 8049 leader_election.cc:302] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b7c73f3fb549dda42748b879cc410f, ac73989afb9041aab439550a6df35e0e; no voters:
I20260929 10:27:19.894234 8316 raft_consensus.cc:2781] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:19.894491 8316 raft_consensus.cc:686] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 1 LEADER]: Becoming Leader. State: Replica: ac73989afb9041aab439550a6df35e0e, State: Running, Role: LEADER
I20260929 10:27:19.894815 8316 consensus_queue.cc:237] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } }
I20260929 10:27:19.898023 7841 catalog_manager.cc:5495] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to ac73989afb9041aab439550a6df35e0e (127.21.33.130). New cstate: current_term: 1 leader_uuid: "ac73989afb9041aab439550a6df35e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac73989afb9041aab439550a6df35e0e" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 43005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 } health_report { overall_health: UNKNOWN } } }
I20260929 10:27:19.929895 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.926630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.21.33.130:43005)
0929 10:27:19.926757 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:19.926762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:19.926777 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:27:19.926849 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:19.926851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:19.926905 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:19.926994 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:19.927217 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.927722 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.928531 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:19.928741 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:19.928813 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:19.929059 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:19.929542 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:19.929662 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:19.929734 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 10:27:19.929768 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 10:27:19.935716 8246 raft_consensus.cc:1261] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 1 FOLLOWER]: Refusing update from remote peer ac73989afb9041aab439550a6df35e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:19.935716 7984 raft_consensus.cc:1261] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 1 FOLLOWER]: Refusing update from remote peer ac73989afb9041aab439550a6df35e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:19.936437 8319 consensus_queue.cc:1035] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51b7c73f3fb549dda42748b879cc410f" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:19.936621 8316 consensus_queue.cc:1035] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e391f2290724eb2bb4b249070962f2a" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 33739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:19.945613 8324 mvcc.cc:205] Tried to move back new op lower bound from 7334615613169815552 to 7334615613011750912. Current Snapshot: MvccSnapshot[applied={T|T < 7334615613169815552}]
I20260929 10:27:19.951526 7837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:19.951709 7837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:19.954145 7837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a802fda66bc482eb472af26f65df804: Sending DeleteTablet for 3 replicas of tablet feec82a5387240da8b55eab12cf7560a
I20260929 10:27:19.954972 7964 tablet_service.cc:1514] Processing DeleteTablet for tablet feec82a5387240da8b55eab12cf7560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:19 UTC) from {username='slave'} at 127.0.0.1:43876
I20260929 10:27:19.955084 8226 tablet_service.cc:1514] Processing DeleteTablet for tablet feec82a5387240da8b55eab12cf7560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:19 UTC) from {username='slave'} at 127.0.0.1:44986
I20260929 10:27:19.955226 8095 tablet_service.cc:1514] Processing DeleteTablet for tablet feec82a5387240da8b55eab12cf7560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:19 UTC) from {username='slave'} at 127.0.0.1:58832
I20260929 10:27:19.955703 8332 tablet_replica.cc:330] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f: stopping tablet replica
I20260929 10:27:19.955843 8333 tablet_replica.cc:330] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: stopping tablet replica
I20260929 10:27:19.956015 8334 tablet_replica.cc:330] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: stopping tablet replica
I20260929 10:27:19.956055 8332 raft_consensus.cc:2229] T feec82a5387240da8b55eab12cf7560a P 51b7c73f3fb549dda42748b879cc410f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:19.956106 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 7902
I20260929 10:27:19.957876 8333 raft_consensus.cc:2229] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:19.957939 8334 raft_consensus.cc:2229] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:27:19.958221 8333 raft_consensus.cc:2258] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:19.958302 8334 raft_consensus.cc:2258] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:19.959719 8333 ts_tablet_manager.cc:1905] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:19.959719 8334 ts_tablet_manager.cc:1905] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:19.962498 8334 ts_tablet_manager.cc:1918] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:19.962487 8333 ts_tablet_manager.cc:1918] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:19.962597 8334 log.cc:1196] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/feec82a5387240da8b55eab12cf7560a
I20260929 10:27:19.962597 8333 log.cc:1196] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/feec82a5387240da8b55eab12cf7560a
W20260929 10:27:19.962733 7827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:34983 (error 108)
I20260929 10:27:19.962924 8334 ts_tablet_manager.cc:1939] T feec82a5387240da8b55eab12cf7560a P ac73989afb9041aab439550a6df35e0e: Deleting consensus metadata
I20260929 10:27:19.962926 8333 ts_tablet_manager.cc:1939] T feec82a5387240da8b55eab12cf7560a P 9e391f2290724eb2bb4b249070962f2a: Deleting consensus metadata
I20260929 10:27:19.963546 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8033
I20260929 10:27:19.964138 7829 catalog_manager.cc:4841] TS 9e391f2290724eb2bb4b249070962f2a (127.21.33.131:33739): tablet feec82a5387240da8b55eab12cf7560a (table test-table [id=db0c603d532a4db39575a216588831b1]) successfully deleted
I20260929 10:27:19.965046 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.964778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:34983 (local address 127.0.0.1:43888)
0929 10:27:19.964908 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.964975 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:34983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 10:27:19.965279 7827 catalog_manager.cc:4584] TS 51b7c73f3fb549dda42748b879cc410f (127.21.33.129:34983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet feec82a5387240da8b55eab12cf7560a: Network error: Client connection negotiation failed: client connection to 127.21.33.129:34983: connect: Connection refused (error 111)
I20260929 10:27:19.969799 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8164
I20260929 10:27:19.971175 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:19.969936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:43005 (local address 127.0.0.1:58846)
0929 10:27:19.971048 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:19.971097 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.130:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1035}
I20260929 10:27:19.975419 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 7811
2026-09-29T10:27:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1059 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:20 dist-test-slave-2rbl krb5kdc[8338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:20 dist-test-slave-2rbl krb5kdc[8338](info): set up 2 sockets
Sep 29 10:27:20 dist-test-slave-2rbl krb5kdc[8338](info): commencing operation
krb5kdc: starting...
W20260929 10:27:21.034101 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:21 dist-test-slave-2rbl krb5kdc[8338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:21.157522 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39315
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:39315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:21.268733 8351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:21.269040 8351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:21.269121 8351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:21.273166 8351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:21.273280 8351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:21.273324 8351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:21.273360 8351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:21.273399 8351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:21.278718 8351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:39315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:21.279804 8351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:21.280864 8351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:21.289592 8359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:21.289563 8357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:21.289589 8356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:21.289664 8351 server_base.cc:1034] running on GCE node
I20260929 10:27:21.290391 8351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:21.291376 8351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:21.292537 8351 hybrid_clock.cc:648] HybridClock initialized: now 1790677641292520 us; error 28 us; skew 500 ppm
Sep 29 10:27:21 dist-test-slave-2rbl krb5kdc[8338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677641, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:21.295262 8351 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:21.296135 8351 webserver.cc:460] Webserver started at http://127.21.33.190:37355/ using document root <none> and password file <none>
I20260929 10:27:21.296721 8351 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:21.296810 8351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:21.297037 8351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:21.299129 8351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "d4654182d64c4e39a964385ce486d485"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "b6137ecab1450666d8a082dfa92f5eb4"
server_key_iv: "443d11835daa1f15f594516b8894aad5"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.299690 8351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "d4654182d64c4e39a964385ce486d485"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "b6137ecab1450666d8a082dfa92f5eb4"
server_key_iv: "443d11835daa1f15f594516b8894aad5"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.303359 8351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:27:21.306073 8366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:21.307287 8351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:21.307420 8351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "d4654182d64c4e39a964385ce486d485"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "b6137ecab1450666d8a082dfa92f5eb4"
server_key_iv: "443d11835daa1f15f594516b8894aad5"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.307525 8351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:21.328552 8351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:21.331430 8351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:21.331629 8351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:21.338739 8351 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:39315
I20260929 10:27:21.338781 8418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:39315 every 8 connection(s)
I20260929 10:27:21.339651 8351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:21.342214 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:21.343955 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8351
I20260929 10:27:21.344084 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:21.344355 21638 external_mini_cluster.cc:1428] Setting key 9c3954e09b6f2c4cf28aa8f58305749e
I20260929 10:27:21.348971 8419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485: Bootstrap starting.
I20260929 10:27:21.351511 8419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:21.352408 8419 log.cc:824] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:27:21 dist-test-slave-2rbl krb5kdc[8338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:21.354615 8419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485: No bootstrap required, opened a new log
I20260929 10:27:21.359575 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:21.345811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.21.33.190:39315)
0929 10:27:21.346251 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:21.346270 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:21.346310 (+ 40us) server_negotiation.cc:407] Connection header received
0929 10:27:21.347123 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:21.347149 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:21.347556 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:21.347921 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:21.348575 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.350020 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.350884 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.351567 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.351730 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:21.354565 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:21.354572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:21.354587 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:21.354617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:21.356509 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:21.356924 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:21.356932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:21.356939 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:21.357115 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:21.357428 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:21.357431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:21.357433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:21.357903 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:21.358117 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:21.358308 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 10:27:21.358621 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":129,"threads_started":1}
I20260929 10:27:21.359387 8419 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4654182d64c4e39a964385ce486d485" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39315 } }
I20260929 10:27:21.359697 8419 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:21.359748 8419 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4654182d64c4e39a964385ce486d485, State: Initialized, Role: FOLLOWER
I20260929 10:27:21.360204 8419 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4654182d64c4e39a964385ce486d485" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39315 } }
I20260929 10:27:21.360363 8419 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:21.360428 8419 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:21.360517 8419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:21.361518 8419 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4654182d64c4e39a964385ce486d485" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39315 } }
I20260929 10:27:21.361889 8419 leader_election.cc:302] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4654182d64c4e39a964385ce486d485; no voters:
I20260929 10:27:21.362190 8419 leader_election.cc:288] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:21.362322 8424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:21.362532 8424 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [term 1 LEADER]: Becoming Leader. State: Replica: d4654182d64c4e39a964385ce486d485, State: Running, Role: LEADER
I20260929 10:27:21.362910 8424 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4654182d64c4e39a964385ce486d485" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39315 } }
I20260929 10:27:21.363430 8419 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:21.364553 8425 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4654182d64c4e39a964385ce486d485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4654182d64c4e39a964385ce486d485" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39315 } } }
I20260929 10:27:21.364547 8426 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4654182d64c4e39a964385ce486d485. Latest consensus state: current_term: 1 leader_uuid: "d4654182d64c4e39a964385ce486d485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4654182d64c4e39a964385ce486d485" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39315 } } }
I20260929 10:27:21.364683 8426 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:21.364683 8425 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:21.365154 8433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:21.368220 8433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:21.373273 8433 catalog_manager.cc:1347] Generated new cluster ID: 9a2d597a19d74eb0afaad0e697881599
I20260929 10:27:21.373340 8433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:21.389016 8433 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:21.390026 8433 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:21.399881 8433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485: Generated new TSK 0
I20260929 10:27:21.400595 8433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.33.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
Entry for principal kudu/127.21.33.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab.
I20260929 10:27:21.433705 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39315
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:21.546452 8445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:21.546741 8445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:21.546828 8445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:21.550985 8445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:21.551075 8445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:21.551168 8445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:27:21.556479 8445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.129.keytab
--principal=kudu/127.21.33.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:21.557646 8445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:21.558686 8445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:21.568481 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:21.568480 8450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:21.568506 8453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:21.568914 8445 server_base.cc:1034] running on GCE node
I20260929 10:27:21.569403 8445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:21.570096 8445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:21.571306 8445 hybrid_clock.cc:648] HybridClock initialized: now 1790677641571269 us; error 46 us; skew 500 ppm
Sep 29 10:27:21 dist-test-slave-2rbl krb5kdc[8338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677641, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:21.574082 8445 init.cc:377] Logged in from keytab as kudu/127.21.33.129@KRBTEST.COM (short username kudu)
I20260929 10:27:21.574900 8445 webserver.cc:460] Webserver started at http://127.21.33.129:36743/ using document root <none> and password file <none>
I20260929 10:27:21.575407 8445 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:21.575464 8445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:21.575625 8445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:21.577646 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "21e03b17887844ce90dad659231c8ef6"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "64a79a176e2ab8e77a9acb018545893a"
server_key_iv: "130a49bec3f9a0cfc7e8b709394fe250"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.578155 8445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "21e03b17887844ce90dad659231c8ef6"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "64a79a176e2ab8e77a9acb018545893a"
server_key_iv: "130a49bec3f9a0cfc7e8b709394fe250"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.582036 8445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:27:21.584295 8460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:21.585538 8445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:21.585659 8445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "21e03b17887844ce90dad659231c8ef6"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "64a79a176e2ab8e77a9acb018545893a"
server_key_iv: "130a49bec3f9a0cfc7e8b709394fe250"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.585759 8445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:21.598843 8445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:21.602159 8445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:21.602380 8445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:21.603024 8445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:21.603925 8445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:21.604001 8445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:21.604069 8445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:21.604138 8445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:21.613317 8445 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:37571
I20260929 10:27:21.613344 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:37571 every 8 connection(s)
I20260929 10:27:21.614218 8445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:27:21.620738 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8445
I20260929 10:27:21.620848 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:27:21.621141 21638 external_mini_cluster.cc:1428] Setting key 4e8db03d440092cd50b0e12baf6fa310
Sep 29 10:27:21 dist-test-slave-2rbl krb5kdc[8338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677641, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.129@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:21.628964 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:21.616010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:45475 (local address 127.21.33.190:39315)
0929 10:27:21.616146 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:21.616150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:21.617000 (+ 850us) server_negotiation.cc:407] Connection header received
0929 10:27:21.618340 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:21.618343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:21.618390 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:21.618471 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:21.619649 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.620199 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.621476 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.621752 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.621841 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:21.624830 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:21.624833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:21.624835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:21.624859 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:21.626441 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:21.627142 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:21.627145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:21.627146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:21.627227 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:21.627675 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:21.627678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:21.627679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:21.627879 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:21.627995 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:21.628728 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 10:27:21.628840 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 10:27:21.630201 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:21.616426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39315 (local address 127.21.33.129:45475)
0929 10:27:21.616850 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:21.616890 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:21.618105 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:21.618635 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:21.618642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:21.619002 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:21.619497 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:21.619510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.620310 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.620313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:21.621359 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:21.621368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.621902 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.621906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:21.622077 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:21.622721 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:21.622742 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:21.624628 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:21.626600 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:21.626610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:21.626631 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:21.627008 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:21.627332 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:21.627338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:21.627340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:21.627545 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:21.628023 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:21.628032 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:21.628409 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 10:27:21.628651 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 10:27:21.629032 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.21.33.130@KRBTEST.COM; defaulting to no policy
I20260929 10:27:21.631584 8574 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39315
I20260929 10:27:21.631800 8574 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:21.632295 8574 heartbeater.cc:511] Master 127.21.33.190:39315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.130@KRBTEST.COM" created.
I20260929 10:27:21.633876 8383 ts_manager.cc:194] Registered new tserver with Master: 21e03b17887844ce90dad659231c8ef6 (127.21.33.129:37571)
I20260929 10:27:21.635007 8383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.129@KRBTEST.COM'} at 127.21.33.129:45475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
Entry for principal kudu/127.21.33.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab.
I20260929 10:27:21.651736 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39315
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:21.760866 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:21.761163 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:21.761276 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:21.765618 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:21.765672 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:21.765738 8579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:27:21.770926 8579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.130.keytab
--principal=kudu/127.21.33.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:21.771894 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:21.772858 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:21.782593 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:21.782645 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:21.782752 8579 server_base.cc:1034] running on GCE node
W20260929 10:27:21.782645 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:21.783334 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:21.783985 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:21.785146 8579 hybrid_clock.cc:648] HybridClock initialized: now 1790677641785130 us; error 30 us; skew 500 ppm
Sep 29 10:27:21 dist-test-slave-2rbl krb5kdc[8338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677641, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:21.787881 8579 init.cc:377] Logged in from keytab as kudu/127.21.33.130@KRBTEST.COM (short username kudu)
I20260929 10:27:21.788829 8579 webserver.cc:460] Webserver started at http://127.21.33.130:36611/ using document root <none> and password file <none>
I20260929 10:27:21.789443 8579 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:21.789506 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:21.789672 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:21.791635 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "2589c781d08d49778638b98791bbec4b"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "51d0cb2300c83ec2bc3397d7c25781e5"
server_key_iv: "a7fae521ecfd2159f29d0f0a72b9a35f"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.792136 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "2589c781d08d49778638b98791bbec4b"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "51d0cb2300c83ec2bc3397d7c25781e5"
server_key_iv: "a7fae521ecfd2159f29d0f0a72b9a35f"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.795749 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:27:21.798300 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:21.799535 8579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:21.799669 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "2589c781d08d49778638b98791bbec4b"
format_stamp: "Formatted at 2026-09-29 10:27:21 on dist-test-slave-2rbl"
server_key: "51d0cb2300c83ec2bc3397d7c25781e5"
server_key_iv: "a7fae521ecfd2159f29d0f0a72b9a35f"
server_key_version: "encryptionkey@0"
I20260929 10:27:21.799787 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:21.824299 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:21.827317 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:21.827524 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:21.828126 8579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:21.828958 8579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:21.829027 8579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:21.829098 8579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:21.829152 8579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:21.837779 8579 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:39335
I20260929 10:27:21.837797 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:39335 every 8 connection(s)
I20260929 10:27:21.838650 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
Sep 29 10:27:21 dist-test-slave-2rbl krb5kdc[8338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677641, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.130@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:21.848197 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8579
I20260929 10:27:21.848333 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:27:21.848629 21638 external_mini_cluster.cc:1428] Setting key 7bfae1092ae214e89619bdfde87dabcf
I20260929 10:27:21.852521 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:21.840480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:44011 (local address 127.21.33.190:39315)
0929 10:27:21.840622 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:21.840626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:21.841498 (+ 872us) server_negotiation.cc:407] Connection header received
0929 10:27:21.842850 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:21.842854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:21.842910 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:21.842990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:21.844137 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.844674 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.845899 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.846125 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.846257 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:21.848866 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:21.848869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:21.848872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:21.848896 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:21.850489 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:21.851002 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:21.851006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:21.851008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:21.851091 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:21.851439 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:21.851443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:21.851445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:21.851633 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:21.851742 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:21.852248 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 10:27:21.852363 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:27:21.853477 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:21.840863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39315 (local address 127.21.33.130:44011)
0929 10:27:21.841356 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:21.841398 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:21.842628 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:21.843158 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:21.843165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:21.843505 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:21.843983 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:21.843996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.844820 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.844823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:21.845769 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:21.845779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:21.846297 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:21.846301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:21.846473 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:21.847002 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:21.847018 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:21.848652 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:21.850607 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:21.850613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:21.850629 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:21.850895 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:21.851190 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:21.851193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:21.851195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:21.851343 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:21.851744 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:21.851754 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:21.852006 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 10:27:21.852201 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 10:27:21.852532 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260929 10:27:21.854802 8708 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39315
I20260929 10:27:21.855042 8708 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:21.855518 8708 heartbeater.cc:511] Master 127.21.33.190:39315 requested a full tablet report, sending...
I20260929 10:27:21.856637 8383 ts_manager.cc:194] Registered new tserver with Master: 2589c781d08d49778638b98791bbec4b (127.21.33.130:39335)
I20260929 10:27:21.857498 8383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.130@KRBTEST.COM'} at 127.21.33.130:44011
WARNING: no policy specified for kudu/127.21.33.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
Entry for principal kudu/127.21.33.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab.
I20260929 10:27:21.880810 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39315
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:46309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:21.991773 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:21.992027 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:21.992098 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:21.996131 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:21.996194 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:21.996253 8713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:27:22.001466 8713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.131.keytab
--principal=kudu/127.21.33.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:22.002462 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:22.003518 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:22.013085 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:22.013091 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:22.013248 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:22.013345 8713 server_base.cc:1034] running on GCE node
I20260929 10:27:22.013816 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:22.014443 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:22.015664 8713 hybrid_clock.cc:648] HybridClock initialized: now 1790677642015645 us; error 42 us; skew 500 ppm
Sep 29 10:27:22 dist-test-slave-2rbl krb5kdc[8338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677642, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:22.018528 8713 init.cc:377] Logged in from keytab as kudu/127.21.33.131@KRBTEST.COM (short username kudu)
I20260929 10:27:22.019398 8713 webserver.cc:460] Webserver started at http://127.21.33.131:37761/ using document root <none> and password file <none>
I20260929 10:27:22.019964 8713 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:22.020097 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:22.020342 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:22.022457 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "41d99546f6b340bd8d310f882b54cc67"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "6b2bf28f0c4984747c380de9978455cc"
server_key_iv: "23a828588fc29321b97d1dbafc7a37b5"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.023056 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "41d99546f6b340bd8d310f882b54cc67"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "6b2bf28f0c4984747c380de9978455cc"
server_key_iv: "23a828588fc29321b97d1dbafc7a37b5"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.026778 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:27:22.029165 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:22.030406 8713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:22.030541 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "41d99546f6b340bd8d310f882b54cc67"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "6b2bf28f0c4984747c380de9978455cc"
server_key_iv: "23a828588fc29321b97d1dbafc7a37b5"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.030648 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:22.055418 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:22.058406 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:22.058625 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:22.059232 8713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:22.060052 8713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:22.060096 8713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:22.060164 8713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:22.060200 8713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:22.069077 8713 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:39383
I20260929 10:27:22.069094 8841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:39383 every 8 connection(s)
I20260929 10:27:22.070003 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:27:22.077755 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8713
I20260929 10:27:22.077909 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:27:22.078218 21638 external_mini_cluster.cc:1428] Setting key 4101d8a52663ae5e561227c3bdae7fe6
Sep 29 10:27:22 dist-test-slave-2rbl krb5kdc[8338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677642, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.131@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:22.083694 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.071817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:34617 (local address 127.21.33.190:39315)
0929 10:27:22.071949 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.071952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.072755 (+ 803us) server_negotiation.cc:407] Connection header received
0929 10:27:22.074201 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.074204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.074265 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.074364 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:22.075484 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.076012 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.077141 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.077423 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.077533 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.080378 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:22.080381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:22.080383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:22.080406 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:22.081918 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:22.082367 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:22.082370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:22.082371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:22.082441 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:22.082735 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:22.082737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:22.082738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:22.082898 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:22.082992 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.083478 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.083580 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:27:22.084530 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.072182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39315 (local address 127.21.33.131:34617)
0929 10:27:22.072615 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.072655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:22.073975 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:22.074536 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:22.074544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:22.074880 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:22.075336 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.075349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.076144 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.076152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.077008 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.077016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.077568 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.077573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.077742 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.078451 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:22.078466 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:22.080228 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:22.082024 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:22.082028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:22.082039 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:22.082282 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:22.082516 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:22.082518 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:22.082520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:22.082650 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:22.083019 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:22.083027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:22.083274 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 10:27:22.083423 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 10:27:22.083669 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260929 10:27:22.085817 8842 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39315
I20260929 10:27:22.086010 8842 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:22.086472 8842 heartbeater.cc:511] Master 127.21.33.190:39315 requested a full tablet report, sending...
I20260929 10:27:22.087486 8383 ts_manager.cc:194] Registered new tserver with Master: 41d99546f6b340bd8d310f882b54cc67 (127.21.33.131:39383)
I20260929 10:27:22.088045 8383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.33.131@KRBTEST.COM'} at 127.21.33.131:34617
I20260929 10:27:22.091997 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:27:22.100745 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.093948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.21.33.190:39315)
0929 10:27:22.094056 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.094059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.094279 (+ 220us) server_negotiation.cc:407] Connection header received
0929 10:27:22.094361 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.094363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.094399 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.094468 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:22.095100 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.095597 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.096446 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.096662 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.096742 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.098078 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:22.098081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:22.098083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:22.098105 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:22.099343 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:22.099728 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:22.099732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:22.099733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:22.099809 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:22.100044 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:22.100046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:22.100047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:22.100263 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:22.100364 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.100545 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.100637 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:27:22.103404 8383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:27:22.106592 8383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:27:22.121176 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.114934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:39383 (local address 127.0.0.1:52308)
0929 10:27:22.115832 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.115884 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:22.116127 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:22.116660 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:22.116666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:22.116689 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:22.116848 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.116857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.118378 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.118381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.119765 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.119774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.120577 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.120581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.120702 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.120740 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 10:27:22.120882 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 10:27:22.120998 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":72,"spinlock_wait_cycles":7040,"thread_start_us":89,"threads_started":1}
I20260929 10:27:22.121354 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.115718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.21.33.129:37571)
0929 10:27:22.116081 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.116095 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.116119 (+ 24us) server_negotiation.cc:407] Connection header received
0929 10:27:22.116746 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.116751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.116922 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.117342 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:22.117444 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.118140 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.119927 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.120779 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.120880 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.121015 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.121073 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.121167 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":74,"threads_started":1}
I20260929 10:27:22.121241 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.115185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37571 (local address 127.0.0.1:58950)
0929 10:27:22.115969 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.115982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:22.116099 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:22.117076 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:22.117081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:22.117099 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:22.117168 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.117172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.118272 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.118278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.119788 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.119797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.120894 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.120897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.120975 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.120994 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 10:27:22.121050 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 10:27:22.121103 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":70,"spinlock_wait_cycles":5376,"thread_start_us":90,"threads_started":1}
I20260929 10:27:22.122218 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.115363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:39335 (local address 127.0.0.1:42110)
0929 10:27:22.116335 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.116347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:22.116424 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:22.116862 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:22.116865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:22.116881 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:22.116987 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.116996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.119527 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.119531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.120685 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.120695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.121833 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.121837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.121918 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.121939 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 10:27:22.122014 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 10:27:22.122081 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":5,"thread_start_us":78,"threads_started":1}
I20260929 10:27:22.122242 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.115876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.21.33.130:39335)
0929 10:27:22.116406 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.116416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.116438 (+ 22us) server_negotiation.cc:407] Connection header received
0929 10:27:22.116526 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.116530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.116734 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.116889 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:22.117084 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.118246 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.120847 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.121704 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.121833 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.121959 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.122037 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.122121 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":127,"threads_started":1}
I20260929 10:27:22.121554 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.115128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.21.33.131:39383)
0929 10:27:22.115587 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.115600 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.115978 (+ 378us) server_negotiation.cc:407] Connection header received
0929 10:27:22.116285 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.116292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.116508 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.116673 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:22.116991 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.118246 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.119883 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.120458 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.120580 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.120748 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.121334 (+ 586us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.121431 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":166,"threads_started":1}
I20260929 10:27:22.124565 8508 tablet_service.cc:1467] Processing CreateTablet for tablet 8a528c4f103b454ead1d265d006c38b7 (DEFAULT_TABLE table=test-table [id=48fddfbc1bdb4e6bacbbdf00aaf0722c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:22.124776 8642 tablet_service.cc:1467] Processing CreateTablet for tablet 8a528c4f103b454ead1d265d006c38b7 (DEFAULT_TABLE table=test-table [id=48fddfbc1bdb4e6bacbbdf00aaf0722c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:22.125670 8508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a528c4f103b454ead1d265d006c38b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:22.125784 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a528c4f103b454ead1d265d006c38b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:22.127671 8776 tablet_service.cc:1467] Processing CreateTablet for tablet 8a528c4f103b454ead1d265d006c38b7 (DEFAULT_TABLE table=test-table [id=48fddfbc1bdb4e6bacbbdf00aaf0722c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:22.128659 8776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a528c4f103b454ead1d265d006c38b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:22.132154 8859 tablet_bootstrap.cc:492] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: Bootstrap starting.
I20260929 10:27:22.133759 8860 tablet_bootstrap.cc:492] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Bootstrap starting.
I20260929 10:27:22.134440 8859 tablet_bootstrap.cc:654] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:22.135315 8859 log.cc:824] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:22.135486 8861 tablet_bootstrap.cc:492] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Bootstrap starting.
I20260929 10:27:22.135916 8860 tablet_bootstrap.cc:654] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:22.137320 8860 log.cc:824] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:22.137501 8859 tablet_bootstrap.cc:492] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: No bootstrap required, opened a new log
I20260929 10:27:22.137699 8859 ts_tablet_manager.cc:1397] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 10:27:22.137871 8861 tablet_bootstrap.cc:654] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:22.138939 8860 tablet_bootstrap.cc:492] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: No bootstrap required, opened a new log
I20260929 10:27:22.139061 8860 ts_tablet_manager.cc:1397] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 10:27:22.139078 8861 log.cc:824] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:22.140965 8861 tablet_bootstrap.cc:492] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: No bootstrap required, opened a new log
I20260929 10:27:22.141134 8861 ts_tablet_manager.cc:1397] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:27:22.142004 8860 raft_consensus.cc:348] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.142261 8860 raft_consensus.cc:374] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:22.142091 8859 raft_consensus.cc:348] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.142333 8860 raft_consensus.cc:729] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2589c781d08d49778638b98791bbec4b, State: Initialized, Role: FOLLOWER
I20260929 10:27:22.142397 8859 raft_consensus.cc:374] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:22.142457 8859 raft_consensus.cc:729] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e03b17887844ce90dad659231c8ef6, State: Initialized, Role: FOLLOWER
I20260929 10:27:22.142855 8860 consensus_queue.cc:260] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.142911 8859 consensus_queue.cc:260] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.143908 8860 ts_tablet_manager.cc:1428] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:27:22.143930 8859 ts_tablet_manager.cc:1428] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 10:27:22.144300 8574 heartbeater.cc:503] Master 127.21.33.190:39315 was elected leader, sending a full tablet report...
I20260929 10:27:22.144300 8708 heartbeater.cc:503] Master 127.21.33.190:39315 was elected leader, sending a full tablet report...
I20260929 10:27:22.145212 8861 raft_consensus.cc:348] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.145496 8861 raft_consensus.cc:374] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:22.146699 8861 raft_consensus.cc:729] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d99546f6b340bd8d310f882b54cc67, State: Initialized, Role: FOLLOWER
I20260929 10:27:22.147205 8861 consensus_queue.cc:260] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.147881 8842 heartbeater.cc:503] Master 127.21.33.190:39315 was elected leader, sending a full tablet report...
I20260929 10:27:22.148191 8861 ts_tablet_manager.cc:1428] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 10:27:22.302739 8867 raft_consensus.cc:482] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:22.302951 8867 raft_consensus.cc:504] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.304225 8867 leader_election.cc:288] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21e03b17887844ce90dad659231c8ef6 (127.21.33.129:37571), 2589c781d08d49778638b98791bbec4b (127.21.33.130:39335)
I20260929 10:27:22.307734 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.304454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:39335 (local address 127.21.33.131:37695)
0929 10:27:22.304576 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.304593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:22.304721 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:22.304980 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:22.304982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:22.305000 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:22.305139 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.305145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.305893 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.305898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.306677 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.306685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.307309 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.307313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.307414 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.307442 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 10:27:22.307529 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 10:27:22.307598 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"spinlock_wait_cycles":768}
I20260929 10:27:22.307732 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.304613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:37695 (local address 127.21.33.130:39335)
0929 10:27:22.304744 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.304750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.304770 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:27:22.304824 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.304827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.304875 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.304959 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:22.305266 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.305782 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.306799 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.307195 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.307318 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.307372 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.307544 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.307610 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 10:27:22.308075 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.304684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:38825 (local address 127.21.33.129:37571)
0929 10:27:22.304828 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.304833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.305012 (+ 179us) server_negotiation.cc:407] Connection header received
0929 10:27:22.305232 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.305238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.305296 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.305374 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:22.305657 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.306149 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.307140 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.307530 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.307654 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.307700 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.307890 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.307969 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:27:22.308399 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.304453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37571 (local address 127.21.33.131:38825)
0929 10:27:22.304916 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.304926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:22.305065 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:22.305396 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:22.305399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:22.305415 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:22.305550 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.305556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.306258 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.306265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.307017 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.307025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.307646 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.307649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.307744 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.307764 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 10:27:22.308227 (+ 463us) client_negotiation.cc:240] Negotiation successful
0929 10:27:22.308298 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":58,"spinlock_wait_cycles":19584,"thread_start_us":122,"threads_started":1}
I20260929 10:27:22.308382 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a528c4f103b454ead1d265d006c38b7" candidate_uuid: "41d99546f6b340bd8d310f882b54cc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2589c781d08d49778638b98791bbec4b" is_pre_election: true
I20260929 10:27:22.308713 8662 raft_consensus.cc:2454] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41d99546f6b340bd8d310f882b54cc67 in term 0.
I20260929 10:27:22.309008 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a528c4f103b454ead1d265d006c38b7" candidate_uuid: "41d99546f6b340bd8d310f882b54cc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e03b17887844ce90dad659231c8ef6" is_pre_election: true
I20260929 10:27:22.309250 8732 leader_election.cc:302] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2589c781d08d49778638b98791bbec4b, 41d99546f6b340bd8d310f882b54cc67; no voters:
I20260929 10:27:22.309352 8528 raft_consensus.cc:2454] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41d99546f6b340bd8d310f882b54cc67 in term 0.
I20260929 10:27:22.309568 8867 raft_consensus.cc:2781] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:27:22.309652 8867 raft_consensus.cc:482] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:27:22.309705 8867 raft_consensus.cc:3037] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:22.310810 8867 raft_consensus.cc:504] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.311149 8867 leader_election.cc:288] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [CANDIDATE]: Term 1 election: Requested vote from peers 21e03b17887844ce90dad659231c8ef6 (127.21.33.129:37571), 2589c781d08d49778638b98791bbec4b (127.21.33.130:39335)
I20260929 10:27:22.311496 8662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a528c4f103b454ead1d265d006c38b7" candidate_uuid: "41d99546f6b340bd8d310f882b54cc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2589c781d08d49778638b98791bbec4b"
I20260929 10:27:22.311511 8528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a528c4f103b454ead1d265d006c38b7" candidate_uuid: "41d99546f6b340bd8d310f882b54cc67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e03b17887844ce90dad659231c8ef6"
I20260929 10:27:22.311632 8528 raft_consensus.cc:3037] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:22.311633 8662 raft_consensus.cc:3037] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:22.312662 8528 raft_consensus.cc:2454] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d99546f6b340bd8d310f882b54cc67 in term 1.
I20260929 10:27:22.312697 8662 raft_consensus.cc:2454] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d99546f6b340bd8d310f882b54cc67 in term 1.
I20260929 10:27:22.312976 8731 leader_election.cc:302] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e03b17887844ce90dad659231c8ef6, 41d99546f6b340bd8d310f882b54cc67; no voters:
I20260929 10:27:22.313139 8867 raft_consensus.cc:2781] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:22.313510 8867 raft_consensus.cc:686] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 1 LEADER]: Becoming Leader. State: Replica: 41d99546f6b340bd8d310f882b54cc67, State: Running, Role: LEADER
I20260929 10:27:22.313835 8867 consensus_queue.cc:237] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } }
I20260929 10:27:22.316833 8382 catalog_manager.cc:5495] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41d99546f6b340bd8d310f882b54cc67 (127.21.33.131). New cstate: current_term: 1 leader_uuid: "41d99546f6b340bd8d310f882b54cc67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d99546f6b340bd8d310f882b54cc67" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 39383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 } health_report { overall_health: UNKNOWN } } }
W20260929 10:27:22.320894 8843 tablet.cc:2369] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 10:27:22.339725 8709 tablet.cc:2369] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:27:22.356060 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.352352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.21.33.131:39383)
0929 10:27:22.352496 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.352500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.352515 (+ 15us) server_negotiation.cc:407] Connection header received
0929 10:27:22.352638 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.352641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.352708 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.352827 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:22.353085 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.353678 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.354601 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.354843 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.354959 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.355275 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:22.355669 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:22.355802 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.355862 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.355918 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 10:27:22.361982 8528 raft_consensus.cc:1261] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 1 FOLLOWER]: Refusing update from remote peer 41d99546f6b340bd8d310f882b54cc67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:22.361982 8662 raft_consensus.cc:1261] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 1 FOLLOWER]: Refusing update from remote peer 41d99546f6b340bd8d310f882b54cc67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:22.362638 8867 consensus_queue.cc:1035] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2589c781d08d49778638b98791bbec4b" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:22.362841 8869 consensus_queue.cc:1035] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21e03b17887844ce90dad659231c8ef6" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 10:27:22.365819 8575 tablet.cc:2369] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:27:22.370451 8872 mvcc.cc:205] Tried to move back new op lower bound from 7334615623107428352 to 7334615622919626752. Current Snapshot: MvccSnapshot[applied={T|T < 7334615623107428352}]
I20260929 10:27:22.371316 8873 mvcc.cc:205] Tried to move back new op lower bound from 7334615623107428352 to 7334615622919626752. Current Snapshot: MvccSnapshot[applied={T|T < 7334615623107428352}]
I20260929 10:27:22.380816 8383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:22.381014 8383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:22.383582 8383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4654182d64c4e39a964385ce486d485: Sending DeleteTablet for 3 replicas of tablet 8a528c4f103b454ead1d265d006c38b7
I20260929 10:27:22.384404 8508 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a528c4f103b454ead1d265d006c38b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:22 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:58950
I20260929 10:27:22.384588 8642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a528c4f103b454ead1d265d006c38b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:22 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:42110
I20260929 10:27:22.384915 8776 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a528c4f103b454ead1d265d006c38b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:22 UTC) from {username='kudu', principal='kudu/127.21.33.190@KRBTEST.COM'} at 127.0.0.1:52308
I20260929 10:27:22.384968 8882 tablet_replica.cc:330] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6: stopping tablet replica
I20260929 10:27:22.384981 8883 tablet_replica.cc:330] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: stopping tablet replica
I20260929 10:27:22.385320 8882 raft_consensus.cc:2229] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:22.385320 8883 raft_consensus.cc:2229] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:22.385656 8883 raft_consensus.cc:2258] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:22.385656 8882 raft_consensus.cc:2258] T 8a528c4f103b454ead1d265d006c38b7 P 21e03b17887844ce90dad659231c8ef6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:22.385783 8884 tablet_replica.cc:330] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: stopping tablet replica
I20260929 10:27:22.386054 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8445
I20260929 10:27:22.388089 8884 raft_consensus.cc:2229] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:27:22.388496 8884 raft_consensus.cc:2258] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:22.388751 8883 ts_tablet_manager.cc:1905] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:22.391016 8883 ts_tablet_manager.cc:1918] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:22.391131 8883 log.cc:1196] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/8a528c4f103b454ead1d265d006c38b7
I20260929 10:27:22.391431 8883 ts_tablet_manager.cc:1939] T 8a528c4f103b454ead1d265d006c38b7 P 2589c781d08d49778638b98791bbec4b: Deleting consensus metadata
I20260929 10:27:22.392627 8370 catalog_manager.cc:4841] TS 2589c781d08d49778638b98791bbec4b (127.21.33.130:39335): tablet 8a528c4f103b454ead1d265d006c38b7 (table test-table [id=48fddfbc1bdb4e6bacbbdf00aaf0722c]) successfully deleted
I20260929 10:27:22.393930 8884 ts_tablet_manager.cc:1905] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 10:27:22.394614 8369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:37571 (error 108)
I20260929 10:27:22.395455 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8579
I20260929 10:27:22.396584 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.395743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:37571 (local address 127.0.0.1:51596)
0929 10:27:22.396419 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.396493 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:37571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583}
I20260929 10:27:22.397006 8884 ts_tablet_manager.cc:1918] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:22.397109 8884 log.cc:1196] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/8a528c4f103b454ead1d265d006c38b7
W20260929 10:27:22.397286 8369 catalog_manager.cc:4584] TS 21e03b17887844ce90dad659231c8ef6 (127.21.33.129:37571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a528c4f103b454ead1d265d006c38b7: Network error: Client connection negotiation failed: client connection to 127.21.33.129:37571: connect: Connection refused (error 111)
I20260929 10:27:22.397481 8884 ts_tablet_manager.cc:1939] T 8a528c4f103b454ead1d265d006c38b7 P 41d99546f6b340bd8d310f882b54cc67: Deleting consensus metadata
I20260929 10:27:22.398691 8368 catalog_manager.cc:4841] TS 41d99546f6b340bd8d310f882b54cc67 (127.21.33.131:39383): tablet 8a528c4f103b454ead1d265d006c38b7 (table test-table [id=48fddfbc1bdb4e6bacbbdf00aaf0722c]) successfully deleted
I20260929 10:27:22.403182 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8713
I20260929 10:27:22.411221 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8351
2026-09-29T10:27:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2436 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T10:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:22Z Disabled control of system clock
I20260929 10:27:22.442142 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39017
--webserver_interface=127.21.33.190
--webserver_port=0
--builtin_ntp_servers=127.21.33.148:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:27:22.551684 8889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:22.552021 8889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:22.552105 8889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:22.556237 8889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:22.556337 8889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:22.556367 8889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:22.556406 8889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:22.556439 8889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:22.561403 8889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:39017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:22.562520 8889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:22.563939 8889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:22.573161 8897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:22.573151 8895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:22.573230 8889 server_base.cc:1034] running on GCE node
W20260929 10:27:22.573151 8894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:22.574041 8889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:22.575193 8889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:22.576393 8889 hybrid_clock.cc:648] HybridClock initialized: now 1790677642576381 us; error 29 us; skew 500 ppm
I20260929 10:27:22.578485 8889 webserver.cc:460] Webserver started at http://127.21.33.190:46413/ using document root <none> and password file <none>
I20260929 10:27:22.579115 8889 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:22.579210 8889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:22.579444 8889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:22.581598 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "663c44ab1a624e4b9dd6044bc415a73b"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "725f69525ae71d77c97912499cdf490f"
server_key_iv: "c4e8d098190dc7cbb81f96bdf533b837"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.582159 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "663c44ab1a624e4b9dd6044bc415a73b"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "725f69525ae71d77c97912499cdf490f"
server_key_iv: "c4e8d098190dc7cbb81f96bdf533b837"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.586593 8889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:27:22.589039 8903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:22.590369 8889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:27:22.590519 8889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "663c44ab1a624e4b9dd6044bc415a73b"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "725f69525ae71d77c97912499cdf490f"
server_key_iv: "c4e8d098190dc7cbb81f96bdf533b837"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.590636 8889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:22.603406 8889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:22.604146 8889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:22.604358 8889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:22.611387 8889 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:39017
I20260929 10:27:22.611393 8955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:39017 every 8 connection(s)
I20260929 10:27:22.612372 8889 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:22.615502 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:22.618165 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8889
I20260929 10:27:22.618310 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:22.618605 21638 external_mini_cluster.cc:1428] Setting key 5875437870cd375de3533863b6f56325
I20260929 10:27:22.621485 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b: Bootstrap starting.
I20260929 10:27:22.623921 8956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:22.624838 8956 log.cc:824] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:22.626977 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b: No bootstrap required, opened a new log
I20260929 10:27:22.628425 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.620132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.21.33.190:39017)
0929 10:27:22.620748 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.620763 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.620800 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:27:22.621519 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.621539 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.621973 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.622261 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:22.622605 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.623561 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.624822 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.625409 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.625530 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.626373 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:22.626379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:22.626395 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:22.626885 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:22.626955 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:22.626981 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:22.627081 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.627237 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.627514 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":131,"threads_started":1}
I20260929 10:27:22.631224 8956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c44ab1a624e4b9dd6044bc415a73b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39017 } }
I20260929 10:27:22.631438 8956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:22.631472 8956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663c44ab1a624e4b9dd6044bc415a73b, State: Initialized, Role: FOLLOWER
I20260929 10:27:22.631875 8956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c44ab1a624e4b9dd6044bc415a73b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39017 } }
I20260929 10:27:22.632002 8956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:22.632050 8956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:22.632120 8956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:22.632987 8956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c44ab1a624e4b9dd6044bc415a73b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39017 } }
I20260929 10:27:22.633343 8956 leader_election.cc:302] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 663c44ab1a624e4b9dd6044bc415a73b; no voters:
I20260929 10:27:22.633606 8956 leader_election.cc:288] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:22.633725 8961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:22.633952 8961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [term 1 LEADER]: Becoming Leader. State: Replica: 663c44ab1a624e4b9dd6044bc415a73b, State: Running, Role: LEADER
I20260929 10:27:22.634241 8961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c44ab1a624e4b9dd6044bc415a73b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39017 } }
I20260929 10:27:22.634817 8956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:22.635828 8963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 663c44ab1a624e4b9dd6044bc415a73b. Latest consensus state: current_term: 1 leader_uuid: "663c44ab1a624e4b9dd6044bc415a73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c44ab1a624e4b9dd6044bc415a73b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39017 } } }
I20260929 10:27:22.635960 8963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:22.636240 8962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "663c44ab1a624e4b9dd6044bc415a73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663c44ab1a624e4b9dd6044bc415a73b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39017 } } }
I20260929 10:27:22.636318 8962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:22.636513 8970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:22.639611 8970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:22.645094 8970 catalog_manager.cc:1347] Generated new cluster ID: 75103b88c3294169b5a3402edde6112f
I20260929 10:27:22.645169 8970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:22.658362 8970 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:22.659256 8970 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:22.666508 8970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b: Generated new TSK 0
I20260929 10:27:22.667155 8970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:22.674604 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.129:0
--local_ip_for_outbound_sockets=127.21.33.129
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39017
--builtin_ntp_servers=127.21.33.148:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 10:27:22.796329 8980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:22.796588 8980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:22.796627 8980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:22.800673 8980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:22.800720 8980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:22.800773 8980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.129
I20260929 10:27:22.805375 8980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.33.129
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.129
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:22.806377 8980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:22.807703 8980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:22.817518 8986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:22.817509 8988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:22.817633 8980 server_base.cc:1034] running on GCE node
W20260929 10:27:22.817507 8985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:22.818168 8980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:22.818797 8980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:22.819962 8980 hybrid_clock.cc:648] HybridClock initialized: now 1790677642819946 us; error 28 us; skew 500 ppm
I20260929 10:27:22.821895 8980 webserver.cc:460] Webserver started at http://127.21.33.129:41293/ using document root <none> and password file <none>
I20260929 10:27:22.822475 8980 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:22.822539 8980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:22.822804 8980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:22.824844 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data/instance:
uuid: "23c9239996f94e8fb129a8754df69b70"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "36b201d81a4aa94c38305a7ae0b8bd1b"
server_key_iv: "52f6c202479230ece0807143b7618a13"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.825430 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance:
uuid: "23c9239996f94e8fb129a8754df69b70"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "36b201d81a4aa94c38305a7ae0b8bd1b"
server_key_iv: "52f6c202479230ece0807143b7618a13"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.829072 8980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 10:27:22.831547 8994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:22.832839 8980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:22.832971 8980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/wal
uuid: "23c9239996f94e8fb129a8754df69b70"
format_stamp: "Formatted at 2026-09-29 10:27:22 on dist-test-slave-2rbl"
server_key: "36b201d81a4aa94c38305a7ae0b8bd1b"
server_key_iv: "52f6c202479230ece0807143b7618a13"
server_key_version: "encryptionkey@0"
I20260929 10:27:22.833084 8980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:22.850898 8980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:22.851610 8980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:22.851804 8980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:22.852363 8980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:22.853330 8980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:22.853404 8980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:22.853474 8980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:22.853519 8980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:22.862713 8980 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.129:41523
I20260929 10:27:22.862741 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.129:41523 every 8 connection(s)
I20260929 10:27:22.863575 8980 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/data/info.pb
I20260929 10:27:22.871681 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 8980
I20260929 10:27:22.871816 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-0/wal/instance
I20260929 10:27:22.872205 21638 external_mini_cluster.cc:1428] Setting key 1c982bf230608366121a7050ca929731
I20260929 10:27:22.873322 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.865386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:55331 (local address 127.21.33.190:39017)
0929 10:27:22.865514 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:22.865518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:22.866645 (+ 1127us) server_negotiation.cc:407] Connection header received
0929 10:27:22.867961 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:22.867964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:22.868009 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:22.868082 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:22.868930 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.869601 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.871115 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.871341 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.871437 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.872784 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:22.872790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:22.872795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:22.872810 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:22.872835 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:22.872843 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:22.872901 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:22.873095 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 10:27:22.873150 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:27:22.874190 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:22.865828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39017 (local address 127.21.33.129:55331)
0929 10:27:22.866484 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:22.866523 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:22.867774 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:22.868216 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:22.868223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:22.868313 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:22.868806 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.868818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.869756 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.869761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.871005 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:22.871015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:22.871522 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:22.871544 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:22.871769 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:22.872464 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:22.872466 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:22.872519 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:22.872610 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:22.872930 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:22.872941 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:22.872947 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 10:27:22.873075 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 10:27:22.873361 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
I20260929 10:27:22.874298 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.130:0
--local_ip_for_outbound_sockets=127.21.33.130
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39017
--builtin_ntp_servers=127.21.33.148:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:27:22.875509 9108 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39017
I20260929 10:27:22.875774 9108 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:22.876289 9108 heartbeater.cc:511] Master 127.21.33.190:39017 requested a full tablet report, sending...
I20260929 10:27:22.877810 8918 ts_manager.cc:194] Registered new tserver with Master: 23c9239996f94e8fb129a8754df69b70 (127.21.33.129:41523)
I20260929 10:27:22.878973 8918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.129:55331
W20260929 10:27:22.985225 9111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:22.985500 9111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:22.985561 9111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:22.989574 9111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:22.989642 9111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:22.989732 9111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.130
I20260929 10:27:22.994499 9111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.33.130
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.130
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:22.995586 9111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:22.996937 9111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:27:23.006978 9111 server_base.cc:1034] running on GCE node
W20260929 10:27:23.006909 9119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:23.006937 9116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:23.006918 9117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:23.007711 9111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:23.008354 9111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:23.009532 9111 hybrid_clock.cc:648] HybridClock initialized: now 1790677643009514 us; error 30 us; skew 500 ppm
I20260929 10:27:23.011286 9111 webserver.cc:460] Webserver started at http://127.21.33.130:41439/ using document root <none> and password file <none>
I20260929 10:27:23.011860 9111 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:23.011951 9111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:23.012180 9111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:23.014420 9111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data/instance:
uuid: "9ccf69f7ace14321b8d96a22c6a15f2c"
format_stamp: "Formatted at 2026-09-29 10:27:23 on dist-test-slave-2rbl"
server_key: "02357e06327b7fc74b551b4521e9e12f"
server_key_iv: "3f371dfd38ac20c4e55e9772b958c131"
server_key_version: "encryptionkey@0"
I20260929 10:27:23.014953 9111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance:
uuid: "9ccf69f7ace14321b8d96a22c6a15f2c"
format_stamp: "Formatted at 2026-09-29 10:27:23 on dist-test-slave-2rbl"
server_key: "02357e06327b7fc74b551b4521e9e12f"
server_key_iv: "3f371dfd38ac20c4e55e9772b958c131"
server_key_version: "encryptionkey@0"
I20260929 10:27:23.018464 9111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:27:23.020824 9125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:23.022024 9111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:23.022181 9111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal
uuid: "9ccf69f7ace14321b8d96a22c6a15f2c"
format_stamp: "Formatted at 2026-09-29 10:27:23 on dist-test-slave-2rbl"
server_key: "02357e06327b7fc74b551b4521e9e12f"
server_key_iv: "3f371dfd38ac20c4e55e9772b958c131"
server_key_version: "encryptionkey@0"
I20260929 10:27:23.022289 9111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:23.035145 9111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:23.035842 9111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:23.036046 9111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:23.036616 9111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:23.037654 9111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:23.037720 9111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:23.037786 9111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:23.037837 9111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:23.046049 9111 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.130:33615
I20260929 10:27:23.046123 9238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.130:33615 every 8 connection(s)
I20260929 10:27:23.046943 9111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/data/info.pb
I20260929 10:27:23.050688 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9111
I20260929 10:27:23.050849 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal/instance
I20260929 10:27:23.051129 21638 external_mini_cluster.cc:1428] Setting key 281f542c185155ed617f316f0bc3cb05
I20260929 10:27:23.052915 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.33.131:0
--local_ip_for_outbound_sockets=127.21.33.131
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39017
--builtin_ntp_servers=127.21.33.148:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 10:27:23.055998 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.048732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.130:51373 (local address 127.21.33.190:39017)
0929 10:27:23.048875 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.048879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.049921 (+ 1042us) server_negotiation.cc:407] Connection header received
0929 10:27:23.051422 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.051426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.051484 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.051563 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:23.052419 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.052953 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.054138 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.054363 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.054473 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.055519 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:23.055523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:23.055526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:23.055538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:23.055565 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:23.055573 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:23.055644 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.055801 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.055861 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 10:27:23.056883 9241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.049137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39017 (local address 127.21.33.130:51373)
0929 10:27:23.049779 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.049816 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:23.051202 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:23.051726 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:23.051733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:23.051815 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:23.052281 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.052293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.053098 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.053101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.054016 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.054026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.054507 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.054512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.054671 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.055275 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:23.055276 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:23.055314 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:23.055371 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:23.055658 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:23.055669 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:23.055679 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 10:27:23.055801 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 10:27:23.056060 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260929 10:27:23.057976 9239 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39017
I20260929 10:27:23.058238 9239 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:23.058873 9239 heartbeater.cc:511] Master 127.21.33.190:39017 requested a full tablet report, sending...
I20260929 10:27:23.059963 8918 ts_manager.cc:194] Registered new tserver with Master: 9ccf69f7ace14321b8d96a22c6a15f2c (127.21.33.130:33615)
I20260929 10:27:23.060506 8918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.130:51373
W20260929 10:27:23.162829 9242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:23.163100 9242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:23.163137 9242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:23.168607 9242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:23.168694 9242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:23.168808 9242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.33.131
I20260929 10:27:23.174671 9242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.33.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.33.131
--webserver_port=0
--tserver_master_addrs=127.21.33.190:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.33.131
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:23.175750 9242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:23.177095 9242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:23.186570 9250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:23.186570 9247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:23.186625 9242 server_base.cc:1034] running on GCE node
W20260929 10:27:23.186578 9248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:23.187160 9242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:23.187767 9242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:23.188921 9242 hybrid_clock.cc:648] HybridClock initialized: now 1790677643188905 us; error 28 us; skew 500 ppm
I20260929 10:27:23.190778 9242 webserver.cc:460] Webserver started at http://127.21.33.131:43253/ using document root <none> and password file <none>
I20260929 10:27:23.191365 9242 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:23.191450 9242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:23.191669 9242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:23.193737 9242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data/instance:
uuid: "2f7d72f1ef284714821dca5d75c125f7"
format_stamp: "Formatted at 2026-09-29 10:27:23 on dist-test-slave-2rbl"
server_key: "a340cf8527de8bc060d507bc9c7d1d3c"
server_key_iv: "8892858fc2baee63695c1fb3a5ff61b0"
server_key_version: "encryptionkey@0"
I20260929 10:27:23.194321 9242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance:
uuid: "2f7d72f1ef284714821dca5d75c125f7"
format_stamp: "Formatted at 2026-09-29 10:27:23 on dist-test-slave-2rbl"
server_key: "a340cf8527de8bc060d507bc9c7d1d3c"
server_key_iv: "8892858fc2baee63695c1fb3a5ff61b0"
server_key_version: "encryptionkey@0"
I20260929 10:27:23.197911 9242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 10:27:23.200248 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:23.201450 9242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:23.201565 9242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal
uuid: "2f7d72f1ef284714821dca5d75c125f7"
format_stamp: "Formatted at 2026-09-29 10:27:23 on dist-test-slave-2rbl"
server_key: "a340cf8527de8bc060d507bc9c7d1d3c"
server_key_iv: "8892858fc2baee63695c1fb3a5ff61b0"
server_key_version: "encryptionkey@0"
I20260929 10:27:23.201668 9242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:23.214353 9242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:23.215001 9242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:23.215186 9242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:23.215742 9242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 10:27:23.216655 9242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 10:27:23.216733 9242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:23.216821 9242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 10:27:23.216869 9242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:23.225400 9242 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.131:38415
I20260929 10:27:23.225431 9369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.131:38415 every 8 connection(s)
I20260929 10:27:23.226382 9242 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/data/info.pb
I20260929 10:27:23.229105 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9242
I20260929 10:27:23.229261 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal/instance
I20260929 10:27:23.229578 21638 external_mini_cluster.cc:1428] Setting key 896ae5af0df4a1ea4aff2d96b6573716
I20260929 10:27:23.235551 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.228042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.131:54325 (local address 127.21.33.190:39017)
0929 10:27:23.228166 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.228170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.229328 (+ 1158us) server_negotiation.cc:407] Connection header received
0929 10:27:23.230954 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.230957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.231013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.231087 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:23.232103 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.232624 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.233740 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.233968 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.234045 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.235090 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:23.235092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:23.235094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:23.235106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:23.235124 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:23.235131 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:23.235217 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.235354 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.235400 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:27:23.236428 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.228406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:39017 (local address 127.21.33.131:54325)
0929 10:27:23.229106 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.229163 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:23.230772 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:23.231273 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:23.231283 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:23.231393 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:23.231986 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.231999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.232721 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.232723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.233647 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.233656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.234085 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.234088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.234232 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.234835 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 10:27:23.234836 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:23.234874 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 10:27:23.234935 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:23.235234 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:23.235241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:23.235245 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 10:27:23.235342 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 10:27:23.235590 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260929 10:27:23.237481 9370 heartbeater.cc:347] Connected to a master server at 127.21.33.190:39017
I20260929 10:27:23.237727 9370 heartbeater.cc:464] Registering TS with master...
I20260929 10:27:23.238219 9370 heartbeater.cc:511] Master 127.21.33.190:39017 requested a full tablet report, sending...
I20260929 10:27:23.239198 8918 ts_manager.cc:194] Registered new tserver with Master: 2f7d72f1ef284714821dca5d75c125f7 (127.21.33.131:38415)
I20260929 10:27:23.239713 8918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.33.131:54325
I20260929 10:27:23.243360 21638 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 10:27:23.248579 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.245328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.21.33.190:39017)
0929 10:27:23.245444 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.245448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.245631 (+ 183us) server_negotiation.cc:407] Connection header received
0929 10:27:23.245718 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.245720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.245754 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.245816 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:23.246047 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.246524 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.247278 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.247475 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.247546 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.248278 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:23.248281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:23.248283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:23.248292 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:23.248306 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:23.248312 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:23.248376 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.248436 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.248479 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:27:23.251245 8918 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 10:27:23.254091 8918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 10:27:23.267946 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.262811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:41523 (local address 127.0.0.1:56332)
0929 10:27:23.263589 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.263613 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:23.263714 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:23.264116 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:23.264119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:23.264135 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:23.264216 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.264222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.265222 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.265229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.266429 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.266438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.267388 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.267391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.267493 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.267531 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 10:27:23.267677 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 10:27:23.267796 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":84,"threads_started":1}
I20260929 10:27:23.267993 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.263222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.21.33.129:41523)
0929 10:27:23.263536 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.263549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.263705 (+ 156us) server_negotiation.cc:407] Connection header received
0929 10:27:23.263812 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.263816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.263979 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.264128 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:23.264307 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.265093 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.266546 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.267245 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.267352 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.267542 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.267717 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.267831 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
I20260929 10:27:23.268222 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.262123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:38415 (local address 127.0.0.1:43802)
0929 10:27:23.262620 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.262668 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:23.262793 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:23.263759 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:23.263767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:23.263792 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:23.263914 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.263919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.265523 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.265526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.266790 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.266799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.267835 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.267838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.267935 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.267958 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 10:27:23.268009 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 10:27:23.268093 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":32,"thread_start_us":78,"threads_started":1}
I20260929 10:27:23.268358 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.262310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33615 (local address 127.0.0.1:51106)
0929 10:27:23.263498 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.263511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:23.263607 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:23.264284 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:23.264289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:23.264307 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:23.264392 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.264398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.265729 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.265733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.266875 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.266884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.267950 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.267955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.268051 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.268077 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 10:27:23.268159 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 10:27:23.268232 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":114,"threads_started":1}
I20260929 10:27:23.268752 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.262917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.21.33.130:33615)
0929 10:27:23.263368 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.263381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.263713 (+ 332us) server_negotiation.cc:407] Connection header received
0929 10:27:23.263793 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.263799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.263979 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.264128 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:23.264516 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.265619 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.266973 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.267616 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.267725 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.267890 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.268514 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.268606 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":127,"threads_started":1}
I20260929 10:27:23.269565 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.262309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.21.33.131:38415)
0929 10:27:23.263179 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.263194 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.263219 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:27:23.263297 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.263304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.263585 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.264038 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:23.264203 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.265375 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.266910 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.267693 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.269055 (+ 1362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.269273 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.269325 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.269410 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":159,"threads_started":1}
I20260929 10:27:23.271039 9033 tablet_service.cc:1467] Processing CreateTablet for tablet f88869ad784d45e8be31135cd6e681de (DEFAULT_TABLE table=test-table [id=a8b673cbf8ef49f28ec52e7c95e585e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:23.271411 9173 tablet_service.cc:1467] Processing CreateTablet for tablet f88869ad784d45e8be31135cd6e681de (DEFAULT_TABLE table=test-table [id=a8b673cbf8ef49f28ec52e7c95e585e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:23.272022 9033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88869ad784d45e8be31135cd6e681de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:23.272434 9173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88869ad784d45e8be31135cd6e681de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:23.272233 9304 tablet_service.cc:1467] Processing CreateTablet for tablet f88869ad784d45e8be31135cd6e681de (DEFAULT_TABLE table=test-table [id=a8b673cbf8ef49f28ec52e7c95e585e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 10:27:23.273152 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88869ad784d45e8be31135cd6e681de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:23.278327 9387 tablet_bootstrap.cc:492] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70: Bootstrap starting.
I20260929 10:27:23.279345 9388 tablet_bootstrap.cc:492] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Bootstrap starting.
I20260929 10:27:23.280347 9389 tablet_bootstrap.cc:492] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Bootstrap starting.
I20260929 10:27:23.280728 9387 tablet_bootstrap.cc:654] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:23.281397 9388 tablet_bootstrap.cc:654] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:23.281625 9387 log.cc:824] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:23.282157 9389 tablet_bootstrap.cc:654] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:23.282186 9388 log.cc:824] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:23.282958 9389 log.cc:824] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:23.283788 9387 tablet_bootstrap.cc:492] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70: No bootstrap required, opened a new log
I20260929 10:27:23.283991 9387 ts_tablet_manager.cc:1397] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 10:27:23.284122 9388 tablet_bootstrap.cc:492] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: No bootstrap required, opened a new log
I20260929 10:27:23.284322 9388 ts_tablet_manager.cc:1397] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:27:23.284814 9389 tablet_bootstrap.cc:492] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: No bootstrap required, opened a new log
I20260929 10:27:23.284987 9389 ts_tablet_manager.cc:1397] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 10:27:23.287462 9387 raft_consensus.cc:348] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.287703 9387 raft_consensus.cc:374] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:23.287763 9387 raft_consensus.cc:729] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c9239996f94e8fb129a8754df69b70, State: Initialized, Role: FOLLOWER
I20260929 10:27:23.288213 9387 consensus_queue.cc:260] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.288556 9388 raft_consensus.cc:348] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.288805 9388 raft_consensus.cc:374] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:23.288853 9388 raft_consensus.cc:729] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccf69f7ace14321b8d96a22c6a15f2c, State: Initialized, Role: FOLLOWER
I20260929 10:27:23.288844 9108 heartbeater.cc:503] Master 127.21.33.190:39017 was elected leader, sending a full tablet report...
I20260929 10:27:23.288976 9389 raft_consensus.cc:348] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.289181 9389 raft_consensus.cc:374] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:23.289212 9387 ts_tablet_manager.cc:1428] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 10:27:23.289261 9389 raft_consensus.cc:729] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7d72f1ef284714821dca5d75c125f7, State: Initialized, Role: FOLLOWER
I20260929 10:27:23.289279 9388 consensus_queue.cc:260] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.289672 9389 consensus_queue.cc:260] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.290037 9239 heartbeater.cc:503] Master 127.21.33.190:39017 was elected leader, sending a full tablet report...
I20260929 10:27:23.290227 9388 ts_tablet_manager.cc:1428] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 10:27:23.290541 9389 ts_tablet_manager.cc:1428] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260929 10:27:23.291115 9370 heartbeater.cc:503] Master 127.21.33.190:39017 was elected leader, sending a full tablet report...
W20260929 10:27:23.297816 9240 tablet.cc:2369] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 10:27:23.300173 9393 raft_consensus.cc:482] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 10:27:23.300343 9393 raft_consensus.cc:504] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.301488 9393 leader_election.cc:288] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f7d72f1ef284714821dca5d75c125f7 (127.21.33.131:38415), 9ccf69f7ace14321b8d96a22c6a15f2c (127.21.33.130:33615)
I20260929 10:27:23.306193 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.301765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:33105 (local address 127.21.33.131:38415)
0929 10:27:23.301895 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.301901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.301914 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:27:23.302062 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.302065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.302109 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.302173 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:23.302483 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.303255 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.304938 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.305583 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.305749 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.305797 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.305978 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.306058 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:27:23.306228 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.301650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.131:38415 (local address 127.21.33.129:33105)
0929 10:27:23.301834 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.301850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:23.301936 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:23.302205 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:23.302208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:23.302233 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:23.302369 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.302375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.303357 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.303359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.304807 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.304818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.305752 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.305756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.305851 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.305873 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 10:27:23.305945 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:27:23.306065 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":87,"spinlock_wait_cycles":4096}
I20260929 10:27:23.306329 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.301827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.33.129:50611 (local address 127.21.33.130:33615)
0929 10:27:23.301945 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.301949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.302226 (+ 277us) server_negotiation.cc:407] Connection header received
0929 10:27:23.302327 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.302329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.302368 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.302425 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 10:27:23.302673 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.303373 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.305044 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.305640 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.305749 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.305797 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.306126 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.306194 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:27:23.306648 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.301719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33615 (local address 127.21.33.129:50611)
0929 10:27:23.302116 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.302127 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:23.302231 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:23.302457 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:23.302460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:23.302480 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 10:27:23.302581 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.302585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.303460 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.303463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.304807 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:23.304818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.305869 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.305872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:23.305987 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.306011 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 10:27:23.306455 (+ 444us) client_negotiation.cc:240] Negotiation successful
0929 10:27:23.306523 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":69,"spinlock_wait_cycles":384,"thread_start_us":119,"threads_started":1}
I20260929 10:27:23.306968 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88869ad784d45e8be31135cd6e681de" candidate_uuid: "23c9239996f94e8fb129a8754df69b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7d72f1ef284714821dca5d75c125f7" is_pre_election: true
I20260929 10:27:23.307297 9324 raft_consensus.cc:2454] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c9239996f94e8fb129a8754df69b70 in term 0.
I20260929 10:27:23.307289 9193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88869ad784d45e8be31135cd6e681de" candidate_uuid: "23c9239996f94e8fb129a8754df69b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" is_pre_election: true
I20260929 10:27:23.307556 9193 raft_consensus.cc:2454] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c9239996f94e8fb129a8754df69b70 in term 0.
I20260929 10:27:23.307862 8996 leader_election.cc:302] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c9239996f94e8fb129a8754df69b70, 2f7d72f1ef284714821dca5d75c125f7; no voters:
I20260929 10:27:23.308153 9393 raft_consensus.cc:2781] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 10:27:23.308234 9393 raft_consensus.cc:482] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 10:27:23.308282 9393 raft_consensus.cc:3037] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:23.309295 9393 raft_consensus.cc:504] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.309687 9393 leader_election.cc:288] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [CANDIDATE]: Term 1 election: Requested vote from peers 2f7d72f1ef284714821dca5d75c125f7 (127.21.33.131:38415), 9ccf69f7ace14321b8d96a22c6a15f2c (127.21.33.130:33615)
I20260929 10:27:23.310007 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88869ad784d45e8be31135cd6e681de" candidate_uuid: "23c9239996f94e8fb129a8754df69b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7d72f1ef284714821dca5d75c125f7"
I20260929 10:27:23.309991 9193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88869ad784d45e8be31135cd6e681de" candidate_uuid: "23c9239996f94e8fb129a8754df69b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c"
I20260929 10:27:23.310137 9324 raft_consensus.cc:3037] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:23.310161 9193 raft_consensus.cc:3037] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:23.311213 9324 raft_consensus.cc:2454] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c9239996f94e8fb129a8754df69b70 in term 1.
I20260929 10:27:23.311271 9193 raft_consensus.cc:2454] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c9239996f94e8fb129a8754df69b70 in term 1.
I20260929 10:27:23.311506 8996 leader_election.cc:302] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c9239996f94e8fb129a8754df69b70, 2f7d72f1ef284714821dca5d75c125f7; no voters:
I20260929 10:27:23.311714 9393 raft_consensus.cc:2781] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:23.311893 9393 raft_consensus.cc:686] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 1 LEADER]: Becoming Leader. State: Replica: 23c9239996f94e8fb129a8754df69b70, State: Running, Role: LEADER
I20260929 10:27:23.312223 9393 consensus_queue.cc:237] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } }
I20260929 10:27:23.315346 8918 catalog_manager.cc:5495] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23c9239996f94e8fb129a8754df69b70 (127.21.33.129). New cstate: current_term: 1 leader_uuid: "23c9239996f94e8fb129a8754df69b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c9239996f94e8fb129a8754df69b70" member_type: VOTER last_known_addr { host: "127.21.33.129" port: 41523 } health_report { overall_health: HEALTHY } } }
I20260929 10:27:23.330516 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.327293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.21.33.129:41523)
0929 10:27:23.327420 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:23.327423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:23.327458 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:27:23.327536 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:23.327539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:23.327599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:23.327689 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:23.327957 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.328501 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.329320 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:23.329531 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:23.329612 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:23.329837 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:23.330235 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:23.330324 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:23.330369 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 10:27:23.330414 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:27:23.336280 9193 raft_consensus.cc:1261] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 1 FOLLOWER]: Refusing update from remote peer 23c9239996f94e8fb129a8754df69b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:23.336280 9324 raft_consensus.cc:1261] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 1 FOLLOWER]: Refusing update from remote peer 23c9239996f94e8fb129a8754df69b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:23.337024 9393 consensus_queue.cc:1035] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f7d72f1ef284714821dca5d75c125f7" member_type: VOTER last_known_addr { host: "127.21.33.131" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:23.337322 9397 consensus_queue.cc:1035] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ccf69f7ace14321b8d96a22c6a15f2c" member_type: VOTER last_known_addr { host: "127.21.33.130" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:23.347280 9402 mvcc.cc:205] Tried to move back new op lower bound from 7334615627098099712 to 7334615627009486848. Current Snapshot: MvccSnapshot[applied={T|T < 7334615627098099712}]
I20260929 10:27:23.353430 8920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:23.353613 8920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 10:27:23.355989 8920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 663c44ab1a624e4b9dd6044bc415a73b: Sending DeleteTablet for 3 replicas of tablet f88869ad784d45e8be31135cd6e681de
I20260929 10:27:23.356773 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet f88869ad784d45e8be31135cd6e681de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:23 UTC) from {username='slave'} at 127.0.0.1:43802
I20260929 10:27:23.356815 9033 tablet_service.cc:1514] Processing DeleteTablet for tablet f88869ad784d45e8be31135cd6e681de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:23 UTC) from {username='slave'} at 127.0.0.1:56332
I20260929 10:27:23.356907 9173 tablet_service.cc:1514] Processing DeleteTablet for tablet f88869ad784d45e8be31135cd6e681de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 10:27:23 UTC) from {username='slave'} at 127.0.0.1:51106
I20260929 10:27:23.357273 9409 tablet_replica.cc:330] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70: stopping tablet replica
I20260929 10:27:23.357573 9410 tablet_replica.cc:330] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: stopping tablet replica
I20260929 10:27:23.357573 9409 raft_consensus.cc:2229] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 1 LEADER]: Raft consensus shutting down.
I20260929 10:27:23.357883 9410 raft_consensus.cc:2229] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:23.357995 9409 raft_consensus.cc:2258] T f88869ad784d45e8be31135cd6e681de P 23c9239996f94e8fb129a8754df69b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:23.358242 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8980
I20260929 10:27:23.358291 9410 raft_consensus.cc:2258] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:23.359867 9411 tablet_replica.cc:330] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: stopping tablet replica
I20260929 10:27:23.360184 9411 raft_consensus.cc:2229] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 10:27:23.360459 9411 raft_consensus.cc:2258] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 10:27:23.362010 9411 ts_tablet_manager.cc:1905] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:23.362177 9410 ts_tablet_manager.cc:1905] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 10:27:23.364490 9411 ts_tablet_manager.cc:1918] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:23.364614 9411 log.cc:1196] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-1/wal/wals/f88869ad784d45e8be31135cd6e681de
I20260929 10:27:23.364759 9410 ts_tablet_manager.cc:1918] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 10:27:23.364856 9410 log.cc:1196] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Deleting WAL directory at /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790677387835684-21638-0/minicluster-data/ts-2/wal/wals/f88869ad784d45e8be31135cd6e681de
I20260929 10:27:23.364952 9411 ts_tablet_manager.cc:1939] T f88869ad784d45e8be31135cd6e681de P 9ccf69f7ace14321b8d96a22c6a15f2c: Deleting consensus metadata
I20260929 10:27:23.365157 9410 ts_tablet_manager.cc:1939] T f88869ad784d45e8be31135cd6e681de P 2f7d72f1ef284714821dca5d75c125f7: Deleting consensus metadata
W20260929 10:27:23.365594 8906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.33.129:41523 (error 108)
I20260929 10:27:23.365799 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9111
I20260929 10:27:23.366494 8905 catalog_manager.cc:4841] TS 2f7d72f1ef284714821dca5d75c125f7 (127.21.33.131:38415): tablet f88869ad784d45e8be31135cd6e681de (table test-table [id=a8b673cbf8ef49f28ec52e7c95e585e8]) successfully deleted
I20260929 10:27:23.367211 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.366956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.129:41523 (local address 127.0.0.1:56340)
0929 10:27:23.367078 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.367140 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.129:41523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 10:27:23.367437 8906 catalog_manager.cc:4584] TS 23c9239996f94e8fb129a8754df69b70 (127.21.33.129:41523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f88869ad784d45e8be31135cd6e681de: Network error: Client connection negotiation failed: client connection to 127.21.33.129:41523: connect: Connection refused (error 111)
I20260929 10:27:23.373116 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9242
I20260929 10:27:23.374722 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:23.373335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.130:33615 (local address 127.0.0.1:51122)
0929 10:27:23.374562 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:23.374627 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.130:33615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1124}
I20260929 10:27:23.378949 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 8889
2026-09-29T10:27:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (967 ms)
[----------] 6 tests from EncryptionPolicyTest (10167 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:23 dist-test-slave-2rbl krb5kdc[9415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:23 dist-test-slave-2rbl krb5kdc[9415](info): set up 2 sockets
Sep 29 10:27:23 dist-test-slave-2rbl krb5kdc[9415](info): commencing operation
krb5kdc: starting...
W20260929 10:27:24.417958 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:24 dist-test-slave-2rbl krb5kdc[9415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:24.543967 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:24.653937 9428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:24.654220 9428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:24.654263 9428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:24.658354 9428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:24.658412 9428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:24.658432 9428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:24.658448 9428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:24.658486 9428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:24.663863 9428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:24.664948 9428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:24.666002 9428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:24.674433 9436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:24.674427 9433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:24.674479 9428 server_base.cc:1034] running on GCE node
W20260929 10:27:24.674427 9434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:24.675244 9428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:24.676270 9428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:24.677421 9428 hybrid_clock.cc:648] HybridClock initialized: now 1790677644677407 us; error 34 us; skew 500 ppm
Sep 29 10:27:24 dist-test-slave-2rbl krb5kdc[9415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:24.680073 9428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:24.680960 9428 webserver.cc:460] Webserver started at http://127.0.0.1:33249/ using document root <none> and password file <none>
I20260929 10:27:24.681599 9428 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:24.681661 9428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:24.681890 9428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:24.683878 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "ef8bde4b0ee541a59e0ba34728688877"
format_stamp: "Formatted at 2026-09-29 10:27:24 on dist-test-slave-2rbl"
server_key: "6d23fd6bc215517321dc247b8bd5e101"
server_key_iv: "dfe2121d9715c62c9ffc6b1cf920d1fa"
server_key_version: "encryptionkey@0"
I20260929 10:27:24.684432 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "ef8bde4b0ee541a59e0ba34728688877"
format_stamp: "Formatted at 2026-09-29 10:27:24 on dist-test-slave-2rbl"
server_key: "6d23fd6bc215517321dc247b8bd5e101"
server_key_iv: "dfe2121d9715c62c9ffc6b1cf920d1fa"
server_key_version: "encryptionkey@0"
I20260929 10:27:24.688144 9428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 10:27:24.690653 9443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:24.691932 9428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:27:24.692099 9428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "ef8bde4b0ee541a59e0ba34728688877"
format_stamp: "Formatted at 2026-09-29 10:27:24 on dist-test-slave-2rbl"
server_key: "6d23fd6bc215517321dc247b8bd5e101"
server_key_iv: "dfe2121d9715c62c9ffc6b1cf920d1fa"
server_key_version: "encryptionkey@0"
I20260929 10:27:24.692217 9428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:24.706089 9428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:24.709982 9428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:24.710168 9428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:24.717422 9428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33281
I20260929 10:27:24.717494 9495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33281 every 8 connection(s)
I20260929 10:27:24.718395 9428 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:24.720139 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9428
I20260929 10:27:24.720279 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:24.720566 21638 external_mini_cluster.cc:1428] Setting key 4709d741e83f7b590bf60e51a1ffcb2b
I20260929 10:27:24.721478 9496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:24.728155 9496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877: Bootstrap starting.
Sep 29 10:27:24 dist-test-slave-2rbl krb5kdc[9415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:27:24.730436 9496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:24.731339 9496 log.cc:824] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:24.733621 9496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877: No bootstrap required, opened a new log
I20260929 10:27:24.735823 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:24.722004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40279 (local address 127.0.0.1:33281)
0929 10:27:24.722422 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:24.722441 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:24.722483 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:24.723377 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:24.723402 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:24.723846 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:24.724215 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:24.724885 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:24.726400 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:24.727306 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:24.727976 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:24.728134 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:24.730779 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:24.730785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:24.730801 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:24.730832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:24.732675 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:24.733176 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:24.733182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:24.733188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:24.733352 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:24.733685 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:24.733689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:24.733690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:24.734176 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:24.734375 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:24.734634 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 10:27:24.734852 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":117,"threads_started":1}
I20260929 10:27:24.738008 9496 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8bde4b0ee541a59e0ba34728688877" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33281 } }
I20260929 10:27:24.738272 9496 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:24.738340 9496 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8bde4b0ee541a59e0ba34728688877, State: Initialized, Role: FOLLOWER
I20260929 10:27:24.738831 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:24.738847 9496 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8bde4b0ee541a59e0ba34728688877" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33281 } }
I20260929 10:27:24.739012 9496 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:24.739076 9496 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:24.739179 9496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:24.740115 9496 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8bde4b0ee541a59e0ba34728688877" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33281 } }
I20260929 10:27:24.740435 9496 leader_election.cc:302] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef8bde4b0ee541a59e0ba34728688877; no voters:
I20260929 10:27:24.740752 9496 leader_election.cc:288] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:24.740831 9505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:24.741037 9505 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [term 1 LEADER]: Becoming Leader. State: Replica: ef8bde4b0ee541a59e0ba34728688877, State: Running, Role: LEADER
I20260929 10:27:24.741448 9505 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8bde4b0ee541a59e0ba34728688877" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33281 } }
I20260929 10:27:24.742060 9496 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:24.743062 9508 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef8bde4b0ee541a59e0ba34728688877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8bde4b0ee541a59e0ba34728688877" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33281 } } }
I20260929 10:27:24.743194 9508 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:24.743512 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef8bde4b0ee541a59e0ba34728688877. Latest consensus state: current_term: 1 leader_uuid: "ef8bde4b0ee541a59e0ba34728688877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8bde4b0ee541a59e0ba34728688877" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33281 } } }
I20260929 10:27:24.743628 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:24.743582 9515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:24.747438 9515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:24.750512 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:24.741085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35239 (local address 127.0.0.1:33281)
0929 10:27:24.741185 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:24.741189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:24.741369 (+ 180us) server_negotiation.cc:407] Connection header received
0929 10:27:24.741541 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:24.741544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:24.741590 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:24.741675 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:24.742366 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:24.742982 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:24.744116 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:24.744447 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:24.744544 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:24.746727 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:24.746731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:24.746733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:24.746758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:24.748468 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:24.749103 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:24.749107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:24.749109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:24.749222 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:24.749603 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:24.749606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:24.749607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:24.749820 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:24.749972 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:24.750248 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 10:27:24.750339 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 10:27:24.753860 9515 catalog_manager.cc:1347] Generated new cluster ID: 4600fc15093146c9bdd5ce6d56ad6ac1
I20260929 10:27:24.753924 9515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:24.778259 9515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:24.779587 9515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:24.791487 9515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef8bde4b0ee541a59e0ba34728688877: Generated new TSK 0
I20260929 10:27:24.792287 9515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:24.803292 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9428
W20260929 10:27:24.808609 9413 connection.cc:535] client connection to 127.0.0.1:33281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33281): Connection reset by peer (error 104)
2026-09-29T10:27:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:24 dist-test-slave-2rbl krb5kdc[9527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:24 dist-test-slave-2rbl krb5kdc[9527](info): set up 2 sockets
Sep 29 10:27:24 dist-test-slave-2rbl krb5kdc[9527](info): commencing operation
krb5kdc: starting...
W20260929 10:27:25.837881 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:25 dist-test-slave-2rbl krb5kdc[9527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:25.976541 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:26.087185 9540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:26.087455 9540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:26.087498 9540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:26.091403 9540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:26.091461 9540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:26.091480 9540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:26.091497 9540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:26.091512 9540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:26.096819 9540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:26.097807 9540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:26.098863 9540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:26.107167 9545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:26.107184 9548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:26.107184 9546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:26.107424 9540 server_base.cc:1034] running on GCE node
I20260929 10:27:26.107926 9540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:26.109031 9540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:26.110209 9540 hybrid_clock.cc:648] HybridClock initialized: now 1790677646110194 us; error 35 us; skew 500 ppm
Sep 29 10:27:26 dist-test-slave-2rbl krb5kdc[9527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:26.112900 9540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:26.113816 9540 webserver.cc:460] Webserver started at http://127.0.0.1:37723/ using document root <none> and password file <none>
I20260929 10:27:26.114320 9540 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:26.114377 9540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:26.114534 9540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:26.116482 9540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "fbc96ae450a541a6952b3a5a72cf3249"
format_stamp: "Formatted at 2026-09-29 10:27:26 on dist-test-slave-2rbl"
server_key: "ee8aa8978cb635cc1ddbc699297c908c"
server_key_iv: "74f2a15268cfb912e37b47b880d36a8e"
server_key_version: "encryptionkey@0"
I20260929 10:27:26.116982 9540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "fbc96ae450a541a6952b3a5a72cf3249"
format_stamp: "Formatted at 2026-09-29 10:27:26 on dist-test-slave-2rbl"
server_key: "ee8aa8978cb635cc1ddbc699297c908c"
server_key_iv: "74f2a15268cfb912e37b47b880d36a8e"
server_key_version: "encryptionkey@0"
I20260929 10:27:26.120504 9540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:27:26.122910 9555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:26.124133 9540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:27:26.124231 9540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "fbc96ae450a541a6952b3a5a72cf3249"
format_stamp: "Formatted at 2026-09-29 10:27:26 on dist-test-slave-2rbl"
server_key: "ee8aa8978cb635cc1ddbc699297c908c"
server_key_iv: "74f2a15268cfb912e37b47b880d36a8e"
server_key_version: "encryptionkey@0"
I20260929 10:27:26.124329 9540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:26.142714 9540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:26.145443 9540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:26.145601 9540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:26.152446 9540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43695
I20260929 10:27:26.152470 9607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43695 every 8 connection(s)
I20260929 10:27:26.153471 9540 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:26.156095 9608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:26.161418 9608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249: Bootstrap starting.
I20260929 10:27:26.163064 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9540
I20260929 10:27:26.163177 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:26.163480 21638 external_mini_cluster.cc:1428] Setting key c4a082bda69c1fe637f1ecb30356baa6
I20260929 10:27:26.164245 9608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:26.165138 9608 log.cc:824] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:26.167282 9608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249: No bootstrap required, opened a new log
I20260929 10:27:26.171500 9608 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc96ae450a541a6952b3a5a72cf3249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43695 } }
I20260929 10:27:26.171769 9608 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:26.171826 9608 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc96ae450a541a6952b3a5a72cf3249, State: Initialized, Role: FOLLOWER
I20260929 10:27:26.172322 9608 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc96ae450a541a6952b3a5a72cf3249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43695 } }
I20260929 10:27:26.172477 9608 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:26.172539 9608 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:26.172632 9608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:26.173427 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:26.164562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58205 (local address 127.0.0.1:43695)
0929 10:27:26.165054 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:26.165073 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:26.165115 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:26.165961 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:26.165985 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:26.166390 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:26.166761 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:26.167167 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:26.168644 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:26.169516 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:26.170326 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:26.170492 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:26.171318 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:26.171323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:26.171340 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:26.171848 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:26.171920 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:26.171950 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:26.172081 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:26.172268 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 10:27:26.172400 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":141,"threads_started":1}
I20260929 10:27:26.173815 9608 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc96ae450a541a6952b3a5a72cf3249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43695 } }
I20260929 10:27:26.174149 9608 leader_election.cc:302] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbc96ae450a541a6952b3a5a72cf3249; no voters:
I20260929 10:27:26.174434 9608 leader_election.cc:288] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:26.174506 9613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:26.174750 9613 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [term 1 LEADER]: Becoming Leader. State: Replica: fbc96ae450a541a6952b3a5a72cf3249, State: Running, Role: LEADER
I20260929 10:27:26.175040 9613 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc96ae450a541a6952b3a5a72cf3249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43695 } }
I20260929 10:27:26.175752 9608 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:26.176366 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:26.176493 9613 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbc96ae450a541a6952b3a5a72cf3249. Latest consensus state: current_term: 1 leader_uuid: "fbc96ae450a541a6952b3a5a72cf3249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc96ae450a541a6952b3a5a72cf3249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43695 } } }
I20260929 10:27:26.176595 9613 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:26.177093 9614 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbc96ae450a541a6952b3a5a72cf3249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc96ae450a541a6952b3a5a72cf3249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43695 } } }
I20260929 10:27:26.177274 9614 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249 [sys.catalog]: This master's current role is: LEADER
W20260929 10:27:26.181329 9631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:26.181408 9631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:26.181481 9621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:26.182318 9621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:26.183214 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:26.178644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50417 (local address 127.0.0.1:43695)
0929 10:27:26.178738 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:26.178742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:26.179178 (+ 436us) server_negotiation.cc:407] Connection header received
0929 10:27:26.179250 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:26.179254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:26.179310 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:26.179389 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:26.179614 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:26.180341 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:26.181526 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:26.181849 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:26.181960 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:26.182791 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:26.182794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:26.182797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:26.182810 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:26.182837 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:26.182849 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:26.182939 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:26.183060 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 10:27:26.183094 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 10:27:26.187970 9621 catalog_manager.cc:1347] Generated new cluster ID: 12c39e09c12641eebc12c475b2d6a4b8
I20260929 10:27:26.188046 9621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:26.204885 9621 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:26.206212 9621 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:26.213889 9621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbc96ae450a541a6952b3a5a72cf3249: Generated new TSK 0
I20260929 10:27:26.214744 9621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:26.221460 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9540
W20260929 10:27:26.227167 9525 connection.cc:535] client connection to 127.0.0.1:43695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43695): Connection reset by peer (error 104)
2026-09-29T10:27:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:26 dist-test-slave-2rbl krb5kdc[9639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:26 dist-test-slave-2rbl krb5kdc[9639](info): set up 2 sockets
Sep 29 10:27:26 dist-test-slave-2rbl krb5kdc[9639](info): commencing operation
krb5kdc: starting...
W20260929 10:27:27.266167 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:27 dist-test-slave-2rbl krb5kdc[9639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:27.388638 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:27.498746 9652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:27.499034 9652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:27.499111 9652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:27.503230 9652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:27.503295 9652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:27.503314 9652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:27.503348 9652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:27.503381 9652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:27.508633 9652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:27.509683 9652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:27.510715 9652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:27.519074 9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:27.519068 9658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:27.519119 9652 server_base.cc:1034] running on GCE node
W20260929 10:27:27.519057 9660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:27.519835 9652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:27.520915 9652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:27.522094 9652 hybrid_clock.cc:648] HybridClock initialized: now 1790677647522060 us; error 44 us; skew 500 ppm
Sep 29 10:27:27 dist-test-slave-2rbl krb5kdc[9639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:27.524878 9652 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:27.525817 9652 webserver.cc:460] Webserver started at http://127.0.0.1:42369/ using document root <none> and password file <none>
I20260929 10:27:27.526358 9652 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:27.526422 9652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:27.526597 9652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:27.528633 9652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "488be086e04d4eb392057e2b5a974dc8"
format_stamp: "Formatted at 2026-09-29 10:27:27 on dist-test-slave-2rbl"
server_key: "ef0ab043a7c00981c8f0b4126ed7b662"
server_key_iv: "347691e6fab9381c7d417b0a07e23f08"
server_key_version: "encryptionkey@0"
I20260929 10:27:27.529259 9652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "488be086e04d4eb392057e2b5a974dc8"
format_stamp: "Formatted at 2026-09-29 10:27:27 on dist-test-slave-2rbl"
server_key: "ef0ab043a7c00981c8f0b4126ed7b662"
server_key_iv: "347691e6fab9381c7d417b0a07e23f08"
server_key_version: "encryptionkey@0"
I20260929 10:27:27.533177 9652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 10:27:27.535851 9667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:27.537268 9652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 10:27:27.537403 9652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "488be086e04d4eb392057e2b5a974dc8"
format_stamp: "Formatted at 2026-09-29 10:27:27 on dist-test-slave-2rbl"
server_key: "ef0ab043a7c00981c8f0b4126ed7b662"
server_key_iv: "347691e6fab9381c7d417b0a07e23f08"
server_key_version: "encryptionkey@0"
I20260929 10:27:27.537525 9652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:27.542129 9652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:27.545315 9652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:27.545492 9652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:27.552342 9652 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35361
I20260929 10:27:27.552361 9720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35361 every 8 connection(s)
I20260929 10:27:27.553359 9652 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:27.554685 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9652
I20260929 10:27:27.554829 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:27.555151 21638 external_mini_cluster.cc:1428] Setting key c5209a698dea23abe2da9e3844fd9c48
I20260929 10:27:27.556615 9721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:27.560386 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:27.556571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42181 (local address 127.0.0.1:35361)
0929 10:27:27.556994 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:27.557008 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:27.557043 (+ 35us) server_negotiation.cc:407] Connection header received
0929 10:27:27.557758 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:27.557774 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:27.558055 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:27.558404 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:27.559221 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:27.559226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:27.559237 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:27.559248 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:27.559300 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:27.559318 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:27.559464 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:27.559533 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 10:27:27.559689 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":138,"threads_started":1}
I20260929 10:27:27.563022 9721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8: Bootstrap starting.
I20260929 10:27:27.563417 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:27.565922 9721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:27.566864 9721 log.cc:824] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:27.567528 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:27.565757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43513 (local address 127.0.0.1:35361)
0929 10:27:27.565903 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:27.565908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:27.566127 (+ 219us) server_negotiation.cc:407] Connection header received
0929 10:27:27.566259 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:27.566261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:27.566294 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:27.566350 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:27.567096 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:27.567098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:27.567100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:27.567110 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:27.567134 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:27.567145 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:27.567214 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:27.567343 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 10:27:27.567383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 10:27:27.568995 9721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8: No bootstrap required, opened a new log
I20260929 10:27:27.572400 9721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488be086e04d4eb392057e2b5a974dc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260929 10:27:27.572618 9721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:27.572677 9721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 488be086e04d4eb392057e2b5a974dc8, State: Initialized, Role: FOLLOWER
I20260929 10:27:27.573052 9721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488be086e04d4eb392057e2b5a974dc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260929 10:27:27.573173 9721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:27.573278 9721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:27.573354 9721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:27.574453 9721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488be086e04d4eb392057e2b5a974dc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260929 10:27:27.574772 9721 leader_election.cc:302] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 488be086e04d4eb392057e2b5a974dc8; no voters:
I20260929 10:27:27.575012 9721 leader_election.cc:288] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:27.575132 9731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:27.575398 9731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 488be086e04d4eb392057e2b5a974dc8, State: Running, Role: LEADER
I20260929 10:27:27.575703 9731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488be086e04d4eb392057e2b5a974dc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260929 10:27:27.576172 9721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:27.577420 9732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "488be086e04d4eb392057e2b5a974dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488be086e04d4eb392057e2b5a974dc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } } }
I20260929 10:27:27.577419 9733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 488be086e04d4eb392057e2b5a974dc8. Latest consensus state: current_term: 1 leader_uuid: "488be086e04d4eb392057e2b5a974dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "488be086e04d4eb392057e2b5a974dc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } } }
I20260929 10:27:27.577630 9733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:27.577656 9732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:27.578027 9743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:27.580989 9743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:27.586855 9743 catalog_manager.cc:1347] Generated new cluster ID: b51e7899d24c45f48cc2c36d1d0ac329
I20260929 10:27:27.586921 9743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:27.602497 9743 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:27.602866 9743 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:27.614843 9743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 488be086e04d4eb392057e2b5a974dc8: Generated new TSK 0
I20260929 10:27:27.615571 9743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:27.629531 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9652
2026-09-29T10:27:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:27 dist-test-slave-2rbl krb5kdc[9752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:27 dist-test-slave-2rbl krb5kdc[9752](info): set up 2 sockets
Sep 29 10:27:27 dist-test-slave-2rbl krb5kdc[9752](info): commencing operation
krb5kdc: starting...
W20260929 10:27:28.658782 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:28 dist-test-slave-2rbl krb5kdc[9752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:28.784855 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:28.894155 9765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:28.894429 9765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:28.894474 9765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:28.898624 9765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:28.898691 9765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:28.898712 9765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:28.898728 9765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:28.898794 9765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:28.904270 9765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:28.905452 9765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:28.906514 9765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:27:28.914752 9765 server_base.cc:1034] running on GCE node
W20260929 10:27:28.914709 9770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:28.914788 9771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:28.914969 9773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:28.915518 9765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:28.916602 9765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:28.917814 9765 hybrid_clock.cc:648] HybridClock initialized: now 1790677648917784 us; error 41 us; skew 500 ppm
Sep 29 10:27:28 dist-test-slave-2rbl krb5kdc[9752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:28.926435 9765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:28.927387 9765 webserver.cc:460] Webserver started at http://127.0.0.1:32817/ using document root <none> and password file <none>
I20260929 10:27:28.927970 9765 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:28.928061 9765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:28.928294 9765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:28.930483 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "4dcba3a1908e43b1b36746c6b7e13555"
format_stamp: "Formatted at 2026-09-29 10:27:28 on dist-test-slave-2rbl"
server_key: "fea9f99a4b1d158aa2901d352f2a46bb"
server_key_iv: "cd56fe7290af908f5cfd14ce6b299b91"
server_key_version: "encryptionkey@0"
I20260929 10:27:28.931056 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "4dcba3a1908e43b1b36746c6b7e13555"
format_stamp: "Formatted at 2026-09-29 10:27:28 on dist-test-slave-2rbl"
server_key: "fea9f99a4b1d158aa2901d352f2a46bb"
server_key_iv: "cd56fe7290af908f5cfd14ce6b299b91"
server_key_version: "encryptionkey@0"
I20260929 10:27:28.934949 9765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 10:27:28.937419 9780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:28.938670 9765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 10:27:28.938817 9765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "4dcba3a1908e43b1b36746c6b7e13555"
format_stamp: "Formatted at 2026-09-29 10:27:28 on dist-test-slave-2rbl"
server_key: "fea9f99a4b1d158aa2901d352f2a46bb"
server_key_iv: "cd56fe7290af908f5cfd14ce6b299b91"
server_key_version: "encryptionkey@0"
I20260929 10:27:28.938930 9765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:28.953709 9765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:28.957072 9765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:28.957326 9765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:28.964640 9765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44229
I20260929 10:27:28.964669 9832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44229 every 8 connection(s)
I20260929 10:27:28.965688 9765 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:28.968425 9833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:28.971071 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9765
I20260929 10:27:28.971235 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:28.971524 21638 external_mini_cluster.cc:1428] Setting key d483d3b061373fa088ba371f05006c91
I20260929 10:27:28.974603 9833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555: Bootstrap starting.
I20260929 10:27:28.977293 9833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:28.978044 9833 log.cc:824] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:28.980165 9833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555: No bootstrap required, opened a new log
Sep 29 10:27:28 dist-test-slave-2rbl krb5kdc[9752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:27:28.984443 9833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcba3a1908e43b1b36746c6b7e13555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44229 } }
I20260929 10:27:28.984714 9833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:28.984790 9833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dcba3a1908e43b1b36746c6b7e13555, State: Initialized, Role: FOLLOWER
I20260929 10:27:28.985324 9833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcba3a1908e43b1b36746c6b7e13555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44229 } }
I20260929 10:27:28.985472 9833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:28.985539 9833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:28.985634 9833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:28.986574 9833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcba3a1908e43b1b36746c6b7e13555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44229 } }
I20260929 10:27:28.986925 9833 leader_election.cc:302] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dcba3a1908e43b1b36746c6b7e13555; no voters:
I20260929 10:27:28.987258 9833 leader_election.cc:288] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:28.987340 9838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:28.987520 9838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [term 1 LEADER]: Becoming Leader. State: Replica: 4dcba3a1908e43b1b36746c6b7e13555, State: Running, Role: LEADER
I20260929 10:27:28.987641 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:28.972932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36979 (local address 127.0.0.1:44229)
0929 10:27:28.973399 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:28.973418 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:28.973460 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:28.974265 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:28.974288 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:28.974737 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:28.975097 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:28.975820 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:28.977314 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:28.978225 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:28.978824 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:28.978985 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:28.981720 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:28.981727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:28.981743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:28.981775 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:28.984429 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:28.984953 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:28.984958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:28.984963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:28.985102 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:28.985427 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:28.985430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:28.985431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:28.985941 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:28.986170 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:28.986386 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 10:27:28.986702 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260929 10:27:28.987883 9838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcba3a1908e43b1b36746c6b7e13555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44229 } }
I20260929 10:27:28.988575 9833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:28.989624 9840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dcba3a1908e43b1b36746c6b7e13555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcba3a1908e43b1b36746c6b7e13555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44229 } } }
I20260929 10:27:28.990316 9840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:28.990198 9839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dcba3a1908e43b1b36746c6b7e13555. Latest consensus state: current_term: 1 leader_uuid: "4dcba3a1908e43b1b36746c6b7e13555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcba3a1908e43b1b36746c6b7e13555" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44229 } } }
I20260929 10:27:28.990664 9839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:28.991130 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 10:27:28.993685 9853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:28.993793 9853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:28.993863 9852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:28.994705 9852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:29.000564 9852 catalog_manager.cc:1347] Generated new cluster ID: bddf2325459c4f11a6cd4573cd0d7843
I20260929 10:27:29.000626 9852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:29.002272 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:28.993710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50891 (local address 127.0.0.1:44229)
0929 10:27:28.993816 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:28.993821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:28.994239 (+ 418us) server_negotiation.cc:407] Connection header received
0929 10:27:28.994376 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:28.994379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:28.994421 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:28.994508 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:28.995195 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:28.995721 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:28.996508 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:28.996833 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:28.996936 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:28.998713 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:28.998716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:28.998719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:28.998743 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:29.000482 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:29.000978 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:29.000981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:29.000983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:29.001085 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:29.001417 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:29.001421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:29.001423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:29.001616 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:29.001737 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:29.001984 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 10:27:29.002101 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 10:27:29.012702 9852 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:29.013983 9852 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:29.019799 9852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dcba3a1908e43b1b36746c6b7e13555: Generated new TSK 0
I20260929 10:27:29.020481 9852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:29.028069 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9765
2026-09-29T10:27:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:29 dist-test-slave-2rbl krb5kdc[9864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:29 dist-test-slave-2rbl krb5kdc[9864](info): set up 2 sockets
Sep 29 10:27:29 dist-test-slave-2rbl krb5kdc[9864](info): commencing operation
krb5kdc: starting...
W20260929 10:27:30.069856 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:30 dist-test-slave-2rbl krb5kdc[9864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:30.193121 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:30.303753 9877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:30.304019 9877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:30.304062 9877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:30.308198 9877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:30.308264 9877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:30.308284 9877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:30.308329 9877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:30.308363 9877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:30.313676 9877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:30.314715 9877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:30.315713 9877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:30.324107 9885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:30.324092 9882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:30.324096 9883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:30.324447 9877 server_base.cc:1034] running on GCE node
I20260929 10:27:30.325029 9877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:30.326169 9877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:30.327356 9877 hybrid_clock.cc:648] HybridClock initialized: now 1790677650327341 us; error 27 us; skew 500 ppm
Sep 29 10:27:30 dist-test-slave-2rbl krb5kdc[9864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:30.330154 9877 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:30.330961 9877 webserver.cc:460] Webserver started at http://127.0.0.1:43109/ using document root <none> and password file <none>
I20260929 10:27:30.331465 9877 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:30.331521 9877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:30.331688 9877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:30.333727 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "8e8cb341e6e74591ac5d5a0978016725"
format_stamp: "Formatted at 2026-09-29 10:27:30 on dist-test-slave-2rbl"
server_key: "4e90895b93a48d5fc19e90dba6a099bb"
server_key_iv: "b5ef93093c05ad3e65e7a7d4a86ac2c2"
server_key_version: "encryptionkey@0"
I20260929 10:27:30.334225 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "8e8cb341e6e74591ac5d5a0978016725"
format_stamp: "Formatted at 2026-09-29 10:27:30 on dist-test-slave-2rbl"
server_key: "4e90895b93a48d5fc19e90dba6a099bb"
server_key_iv: "b5ef93093c05ad3e65e7a7d4a86ac2c2"
server_key_version: "encryptionkey@0"
I20260929 10:27:30.337700 9877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:27:30.339959 9892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:30.341094 9877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:30.341243 9877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "8e8cb341e6e74591ac5d5a0978016725"
format_stamp: "Formatted at 2026-09-29 10:27:30 on dist-test-slave-2rbl"
server_key: "4e90895b93a48d5fc19e90dba6a099bb"
server_key_iv: "b5ef93093c05ad3e65e7a7d4a86ac2c2"
server_key_version: "encryptionkey@0"
I20260929 10:27:30.341341 9877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:30.380225 9877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:30.383137 9877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:30.383296 9877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:30.390131 9877 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41931
I20260929 10:27:30.390162 9944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41931 every 8 connection(s)
I20260929 10:27:30.391057 9877 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:30.393826 9945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:30.399276 9945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725: Bootstrap starting.
I20260929 10:27:30.399875 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9877
I20260929 10:27:30.400020 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:30.400295 21638 external_mini_cluster.cc:1428] Setting key 64baa371b98ea775ebb4baf18c8ab391
I20260929 10:27:30.401974 9945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:30.402689 9945 log.cc:824] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:30.404531 9945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725: No bootstrap required, opened a new log
I20260929 10:27:30.408380 9945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8cb341e6e74591ac5d5a0978016725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41931 } }
I20260929 10:27:30.408623 9945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:30.408705 9945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8cb341e6e74591ac5d5a0978016725, State: Initialized, Role: FOLLOWER
I20260929 10:27:30.409174 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:30.401603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35811 (local address 127.0.0.1:41931)
0929 10:27:30.402014 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:30.402029 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:30.402065 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:27:30.402729 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:30.402746 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:30.403196 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:30.403518 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:30.403926 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:30.404933 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:30.405875 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:30.406354 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:30.406534 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:30.407364 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:30.407369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:30.407382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:30.407765 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:30.407817 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:30.407839 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:30.407947 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:30.408073 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 10:27:30.408287 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":138,"threads_started":1}
I20260929 10:27:30.409135 9945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8cb341e6e74591ac5d5a0978016725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41931 } }
I20260929 10:27:30.409345 9945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:30.409439 9945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:30.409543 9945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:30.410571 9945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8cb341e6e74591ac5d5a0978016725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41931 } }
I20260929 10:27:30.410900 9945 leader_election.cc:302] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e8cb341e6e74591ac5d5a0978016725; no voters:
I20260929 10:27:30.411253 9945 leader_election.cc:288] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:30.411336 9950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:30.411559 9950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [term 1 LEADER]: Becoming Leader. State: Replica: 8e8cb341e6e74591ac5d5a0978016725, State: Running, Role: LEADER
I20260929 10:27:30.411928 9950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8cb341e6e74591ac5d5a0978016725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41931 } }
I20260929 10:27:30.412096 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:30.412525 9945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:30.413067 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e8cb341e6e74591ac5d5a0978016725. Latest consensus state: current_term: 1 leader_uuid: "8e8cb341e6e74591ac5d5a0978016725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8cb341e6e74591ac5d5a0978016725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41931 } } }
I20260929 10:27:30.413288 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:30.413522 9952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e8cb341e6e74591ac5d5a0978016725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8cb341e6e74591ac5d5a0978016725" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41931 } } }
I20260929 10:27:30.413628 9952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725 [sys.catalog]: This master's current role is: LEADER
W20260929 10:27:30.417948 9969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:30.418030 9969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:30.418107 9960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:30.418995 9960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:30.419655 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:30.414522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48341 (local address 127.0.0.1:41931)
0929 10:27:30.414636 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:30.414640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:30.415205 (+ 565us) server_negotiation.cc:407] Connection header received
0929 10:27:30.415494 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:30.415497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:30.415554 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:30.415643 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:30.415926 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:30.416668 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:30.417833 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:30.418229 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:30.418335 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:30.419209 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:30.419213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:30.419216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:30.419229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:30.419257 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:30.419271 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:30.419343 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:30.419461 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 10:27:30.419515 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 10:27:30.425735 9960 catalog_manager.cc:1347] Generated new cluster ID: cea499bda8374e6aa20b1b52b17fc3b7
I20260929 10:27:30.425832 9960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:30.439147 9960 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:30.440433 9960 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:30.450341 9960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e8cb341e6e74591ac5d5a0978016725: Generated new TSK 0
I20260929 10:27:30.451102 9960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:30.458137 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9877
2026-09-29T10:27:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:30 dist-test-slave-2rbl krb5kdc[9976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:30 dist-test-slave-2rbl krb5kdc[9976](info): set up 2 sockets
Sep 29 10:27:30 dist-test-slave-2rbl krb5kdc[9976](info): commencing operation
krb5kdc: starting...
W20260929 10:27:31.517939 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:31 dist-test-slave-2rbl krb5kdc[9976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:31.644029 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:31.755715 9989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:31.755985 9989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:31.756028 9989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:31.760129 9989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:31.760195 9989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:31.760215 9989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:31.760232 9989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:31.760247 9989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:31.765602 9989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:31.766644 9989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:31.767655 9989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:31.776316 9997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:31.776310 9994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:31.776309 9995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:31.776767 9989 server_base.cc:1034] running on GCE node
I20260929 10:27:31.777469 9989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:31.778622 9989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:31.779783 9989 hybrid_clock.cc:648] HybridClock initialized: now 1790677651779750 us; error 44 us; skew 500 ppm
Sep 29 10:27:31 dist-test-slave-2rbl krb5kdc[9976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:31.782614 9989 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:31.783569 9989 webserver.cc:460] Webserver started at http://127.0.0.1:32875/ using document root <none> and password file <none>
I20260929 10:27:31.784158 9989 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:31.784251 9989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:31.784478 9989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:31.786600 9989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "35374164c1cf4cfba3e37d70648681c7"
format_stamp: "Formatted at 2026-09-29 10:27:31 on dist-test-slave-2rbl"
server_key: "fdcf191e01df021a56d276a98cd0a16d"
server_key_iv: "2e90b070a41a7d67df8f1aeb8307ddbf"
server_key_version: "encryptionkey@0"
I20260929 10:27:31.787271 9989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "35374164c1cf4cfba3e37d70648681c7"
format_stamp: "Formatted at 2026-09-29 10:27:31 on dist-test-slave-2rbl"
server_key: "fdcf191e01df021a56d276a98cd0a16d"
server_key_iv: "2e90b070a41a7d67df8f1aeb8307ddbf"
server_key_version: "encryptionkey@0"
I20260929 10:27:31.791218 9989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:27:31.793761 10004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:31.795101 9989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:31.795274 9989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "35374164c1cf4cfba3e37d70648681c7"
format_stamp: "Formatted at 2026-09-29 10:27:31 on dist-test-slave-2rbl"
server_key: "fdcf191e01df021a56d276a98cd0a16d"
server_key_iv: "2e90b070a41a7d67df8f1aeb8307ddbf"
server_key_version: "encryptionkey@0"
I20260929 10:27:31.795408 9989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:31.799993 9989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:31.803097 9989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:31.803318 9989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:31.810554 9989 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40237
I20260929 10:27:31.810575 10056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40237 every 8 connection(s)
I20260929 10:27:31.811560 9989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:31.814431 10057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:31.819864 10057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7: Bootstrap starting.
I20260929 10:27:31.820236 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 9989
I20260929 10:27:31.820372 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:31.820681 21638 external_mini_cluster.cc:1428] Setting key d7e533342bf528307cf85c83a6fa8b47
I20260929 10:27:31.822781 10057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:31.823556 10057 log.cc:824] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:31.825830 10057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7: No bootstrap required, opened a new log
I20260929 10:27:31.826220 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:31.822041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35903 (local address 127.0.0.1:40237)
0929 10:27:31.822521 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:31.822537 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:31.822575 (+ 38us) server_negotiation.cc:407] Connection header received
0929 10:27:31.823339 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:31.823359 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:31.823702 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:31.824054 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:31.824802 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:31.824808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:31.824823 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:31.824834 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:31.824896 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:31.824921 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:31.825035 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:31.825104 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 10:27:31.825299 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":130,"threads_started":1}
I20260929 10:27:31.829277 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:31.830183 10057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35374164c1cf4cfba3e37d70648681c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260929 10:27:31.830474 10057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:31.830531 10057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35374164c1cf4cfba3e37d70648681c7, State: Initialized, Role: FOLLOWER
I20260929 10:27:31.831030 10057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35374164c1cf4cfba3e37d70648681c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260929 10:27:31.831197 10057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:31.831269 10057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:31.831379 10057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:31.832705 10057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35374164c1cf4cfba3e37d70648681c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260929 10:27:31.833101 10057 leader_election.cc:302] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35374164c1cf4cfba3e37d70648681c7; no voters:
I20260929 10:27:31.833376 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:31.831646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51507 (local address 127.0.0.1:40237)
0929 10:27:31.831783 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:31.831787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:31.832062 (+ 275us) server_negotiation.cc:407] Connection header received
0929 10:27:31.832129 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:31.832132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:31.832166 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:31.832234 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:31.832934 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:31.832937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:31.832939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:31.832951 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:31.832978 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:31.832990 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:31.833076 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:31.833181 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 10:27:31.833266 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:27:31.833508 10057 leader_election.cc:288] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:31.834610 10067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:31.834856 10067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [term 1 LEADER]: Becoming Leader. State: Replica: 35374164c1cf4cfba3e37d70648681c7, State: Running, Role: LEADER
I20260929 10:27:31.834928 10057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:31.835223 10067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35374164c1cf4cfba3e37d70648681c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260929 10:27:31.836984 10068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35374164c1cf4cfba3e37d70648681c7. Latest consensus state: current_term: 1 leader_uuid: "35374164c1cf4cfba3e37d70648681c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35374164c1cf4cfba3e37d70648681c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260929 10:27:31.837337 10068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:31.837596 10069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35374164c1cf4cfba3e37d70648681c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35374164c1cf4cfba3e37d70648681c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260929 10:27:31.837751 10069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7 [sys.catalog]: This master's current role is: LEADER
W20260929 10:27:31.839632 10081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:31.839699 10081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:31.839776 10082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:31.840408 10082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:31.846021 10082 catalog_manager.cc:1347] Generated new cluster ID: 8e6c679327f844219cf95cdc3d3186c2
I20260929 10:27:31.846094 10082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:31.854802 10082 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:31.855100 10082 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:31.860101 10082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35374164c1cf4cfba3e37d70648681c7: Generated new TSK 0
I20260929 10:27:31.860749 10082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:31.871330 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 9989
2026-09-29T10:27:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:31 dist-test-slave-2rbl krb5kdc[10088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:31 dist-test-slave-2rbl krb5kdc[10088](info): set up 2 sockets
Sep 29 10:27:31 dist-test-slave-2rbl krb5kdc[10088](info): commencing operation
krb5kdc: starting...
W20260929 10:27:32.910019 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:32 dist-test-slave-2rbl krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:33.036463 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:33.143865 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:33.144184 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:33.144268 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:33.148348 10101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:33.148442 10101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:33.148480 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:33.148541 10101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:33.148588 10101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:33.153927 10101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:33.155005 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:33.156038 10101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:33.164487 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:33.164629 10101 server_base.cc:1034] running on GCE node
W20260929 10:27:33.164650 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:33.164497 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:33.165994 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:33.167634 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:33.168833 10101 hybrid_clock.cc:648] HybridClock initialized: now 1790677653168812 us; error 40 us; skew 500 ppm
Sep 29 10:27:33 dist-test-slave-2rbl krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:33.172672 10101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:33.173754 10101 webserver.cc:460] Webserver started at http://127.0.0.1:34883/ using document root <none> and password file <none>
I20260929 10:27:33.174470 10101 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:33.174561 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:33.174777 10101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:33.176807 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "2a9436d9ba3e48288ca7ff7810ba5b77"
format_stamp: "Formatted at 2026-09-29 10:27:33 on dist-test-slave-2rbl"
server_key: "a83493c243a77f952283f0fd2d43a289"
server_key_iv: "feed61b55d6a3fd09adfe05830076bd2"
server_key_version: "encryptionkey@0"
I20260929 10:27:33.177440 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "2a9436d9ba3e48288ca7ff7810ba5b77"
format_stamp: "Formatted at 2026-09-29 10:27:33 on dist-test-slave-2rbl"
server_key: "a83493c243a77f952283f0fd2d43a289"
server_key_iv: "feed61b55d6a3fd09adfe05830076bd2"
server_key_version: "encryptionkey@0"
I20260929 10:27:33.181286 10101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:27:33.183811 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:33.185086 10101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:33.185281 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "2a9436d9ba3e48288ca7ff7810ba5b77"
format_stamp: "Formatted at 2026-09-29 10:27:33 on dist-test-slave-2rbl"
server_key: "a83493c243a77f952283f0fd2d43a289"
server_key_iv: "feed61b55d6a3fd09adfe05830076bd2"
server_key_version: "encryptionkey@0"
I20260929 10:27:33.185408 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:33.199568 10101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:33.202559 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:33.202764 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:33.209931 10101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46821
I20260929 10:27:33.209945 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46821 every 8 connection(s)
I20260929 10:27:33.210958 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:33.212667 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10101
I20260929 10:27:33.212809 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:33.213121 21638 external_mini_cluster.cc:1428] Setting key 821eb9e8698d55bf08a9dad7076988a3
I20260929 10:27:33.214170 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:33.219744 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77: Bootstrap starting.
Sep 29 10:27:33 dist-test-slave-2rbl krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:27:33.222154 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:33.223009 10169 log.cc:824] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:33.225314 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77: No bootstrap required, opened a new log
I20260929 10:27:33.227783 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:33.214528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:56761 (local address 127.0.0.1:46821)
0929 10:27:33.214959 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:33.214974 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:33.215008 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:27:33.215665 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:33.215682 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:33.216015 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:33.216350 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:33.217142 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:33.218189 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:33.219040 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:33.219510 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:33.219708 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:33.222219 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:33.222227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:33.222247 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:33.222298 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:33.224372 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:33.224926 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:33.224934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:33.224940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:33.225082 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:33.225523 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:33.225526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:33.225528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:33.226036 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:33.226249 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:33.226528 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 10:27:33.226753 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":144,"threads_started":1}
I20260929 10:27:33.230005 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46821 } }
I20260929 10:27:33.230252 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:33.230366 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9436d9ba3e48288ca7ff7810ba5b77, State: Initialized, Role: FOLLOWER
I20260929 10:27:33.230870 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46821 } }
I20260929 10:27:33.230962 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:33.231040 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:33.231107 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:33.231202 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:33.232199 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46821 } }
I20260929 10:27:33.232563 10169 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9436d9ba3e48288ca7ff7810ba5b77; no voters:
I20260929 10:27:33.232893 10169 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:33.233034 10178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:33.233346 10178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [term 1 LEADER]: Becoming Leader. State: Replica: 2a9436d9ba3e48288ca7ff7810ba5b77, State: Running, Role: LEADER
I20260929 10:27:33.233727 10178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46821 } }
I20260929 10:27:33.234318 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:33.235044 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9436d9ba3e48288ca7ff7810ba5b77. Latest consensus state: current_term: 1 leader_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46821 } } }
I20260929 10:27:33.235756 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:33.235646 10180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9436d9ba3e48288ca7ff7810ba5b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46821 } } }
I20260929 10:27:33.236037 10180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77 [sys.catalog]: This master's current role is: LEADER
W20260929 10:27:33.239698 10194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:33.239825 10194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:33.239902 10190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:33.240876 10190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:33.242671 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:33.233078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:40513 (local address 127.0.0.1:46821)
0929 10:27:33.233244 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:33.233248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:33.233475 (+ 227us) server_negotiation.cc:407] Connection header received
0929 10:27:33.233669 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:33.233673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:33.233725 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:33.233812 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:33.234613 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:33.235445 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:33.236840 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:33.237161 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:33.237355 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:33.238963 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:33.238967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:33.238970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:33.238995 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:33.240731 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:33.241312 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:33.241315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:33.241317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:33.241403 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:33.241779 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:33.241782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:33.241783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:33.241988 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:33.242137 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:33.242429 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 10:27:33.242527 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 10:27:33.246654 10190 catalog_manager.cc:1347] Generated new cluster ID: 9c25a9541c6f433d8f656e7723ea55ed
I20260929 10:27:33.246713 10190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:33.260113 10190 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:33.261427 10190 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:33.270898 10190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a9436d9ba3e48288ca7ff7810ba5b77: Generated new TSK 0
I20260929 10:27:33.271756 10190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:33.278185 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10101
W20260929 10:27:33.283502 10086 connection.cc:535] client connection to 127.0.0.1:46821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46821): Connection reset by peer (error 104)
2026-09-29T10:27:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:33 dist-test-slave-2rbl krb5kdc[10200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:33 dist-test-slave-2rbl krb5kdc[10200](info): set up 2 sockets
Sep 29 10:27:33 dist-test-slave-2rbl krb5kdc[10200](info): commencing operation
krb5kdc: starting...
W20260929 10:27:34.325870 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:34 dist-test-slave-2rbl krb5kdc[10200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:34.451815 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:34.562471 10213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:34.562744 10213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:34.562790 10213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:34.566675 10213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:34.566733 10213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:34.566756 10213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:34.566776 10213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:34.566860 10213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:34.572116 10213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:34.573140 10213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:34.574172 10213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:34.582348 10218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:34.582376 10219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:34.582454 10213 server_base.cc:1034] running on GCE node
W20260929 10:27:34.582350 10221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:34.583155 10213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:34.584201 10213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:34.585389 10213 hybrid_clock.cc:648] HybridClock initialized: now 1790677654585363 us; error 37 us; skew 500 ppm
Sep 29 10:27:34 dist-test-slave-2rbl krb5kdc[10200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:34.591984 10213 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:34.592836 10213 webserver.cc:460] Webserver started at http://127.0.0.1:35443/ using document root <none> and password file <none>
I20260929 10:27:34.593425 10213 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:34.593489 10213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:34.593654 10213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:34.595633 10213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "4af189bcdbfe412a9679d3fb79576932"
format_stamp: "Formatted at 2026-09-29 10:27:34 on dist-test-slave-2rbl"
server_key: "4aa1cd4d6799657acd5512b0c9af6862"
server_key_iv: "458da1ca1d1760bddae18fee2aa5c3c3"
server_key_version: "encryptionkey@0"
I20260929 10:27:34.596143 10213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "4af189bcdbfe412a9679d3fb79576932"
format_stamp: "Formatted at 2026-09-29 10:27:34 on dist-test-slave-2rbl"
server_key: "4aa1cd4d6799657acd5512b0c9af6862"
server_key_iv: "458da1ca1d1760bddae18fee2aa5c3c3"
server_key_version: "encryptionkey@0"
I20260929 10:27:34.599807 10213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:27:34.602159 10228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:34.603360 10213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:34.603458 10213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "4af189bcdbfe412a9679d3fb79576932"
format_stamp: "Formatted at 2026-09-29 10:27:34 on dist-test-slave-2rbl"
server_key: "4aa1cd4d6799657acd5512b0c9af6862"
server_key_iv: "458da1ca1d1760bddae18fee2aa5c3c3"
server_key_version: "encryptionkey@0"
I20260929 10:27:34.603542 10213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:34.621850 10213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:34.624787 10213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:34.624943 10213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:34.631925 10213 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43493
I20260929 10:27:34.631944 10280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43493 every 8 connection(s)
I20260929 10:27:34.632884 10213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:34.635553 10281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:34.638085 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10213
I20260929 10:27:34.638221 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:34.638494 21638 external_mini_cluster.cc:1428] Setting key 608be7674db34f50e77f389ae3854248
I20260929 10:27:34.641665 10281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932: Bootstrap starting.
I20260929 10:27:34.644264 10281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:34.645073 10281 log.cc:824] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:34.647240 10281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932: No bootstrap required, opened a new log
I20260929 10:27:34.648564 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:34.639886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:46977 (local address 127.0.0.1:43493)
0929 10:27:34.640284 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:34.640302 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:34.640347 (+ 45us) server_negotiation.cc:407] Connection header received
0929 10:27:34.641159 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:34.641181 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:34.641632 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:34.641987 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:34.642391 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:34.643861 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:34.644761 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:34.645476 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:34.645619 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:34.646588 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:34.646594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:34.646624 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:34.647098 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:34.647162 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:34.647190 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:34.647330 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:34.647537 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 10:27:34.647689 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":96,"threads_started":1}
I20260929 10:27:34.651381 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:34.651449 10281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af189bcdbfe412a9679d3fb79576932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260929 10:27:34.651686 10281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:34.651757 10281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af189bcdbfe412a9679d3fb79576932, State: Initialized, Role: FOLLOWER
I20260929 10:27:34.652168 10281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af189bcdbfe412a9679d3fb79576932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260929 10:27:34.652288 10281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:34.652367 10281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:34.652477 10281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:34.653455 10281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af189bcdbfe412a9679d3fb79576932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260929 10:27:34.653812 10281 leader_election.cc:302] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4af189bcdbfe412a9679d3fb79576932; no voters:
I20260929 10:27:34.654186 10281 leader_election.cc:288] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:34.654280 10291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:34.654492 10291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [term 1 LEADER]: Becoming Leader. State: Replica: 4af189bcdbfe412a9679d3fb79576932, State: Running, Role: LEADER
I20260929 10:27:34.654810 10291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af189bcdbfe412a9679d3fb79576932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260929 10:27:34.655431 10281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:34.655949 10291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4af189bcdbfe412a9679d3fb79576932. Latest consensus state: current_term: 1 leader_uuid: "4af189bcdbfe412a9679d3fb79576932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af189bcdbfe412a9679d3fb79576932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } } }
I20260929 10:27:34.656061 10291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:34.656454 10296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:34.656749 10292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4af189bcdbfe412a9679d3fb79576932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af189bcdbfe412a9679d3fb79576932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } } }
I20260929 10:27:34.656827 10292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:34.659196 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:34.653543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:50999 (local address 127.0.0.1:43493)
0929 10:27:34.653650 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:34.653653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:34.653840 (+ 187us) server_negotiation.cc:407] Connection header received
0929 10:27:34.653972 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:34.653976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:34.654024 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:34.654094 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:34.654442 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:34.655173 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:34.656314 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:34.656671 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:34.656769 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:34.658676 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:34.658680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:34.658683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:34.658695 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:34.658718 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:34.658730 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:34.658844 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:34.659025 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 10:27:34.659057 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 10:27:34.660009 10296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:34.665524 10296 catalog_manager.cc:1347] Generated new cluster ID: 54ca0acdc59e42559306de41758ddf52
I20260929 10:27:34.665591 10296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:34.693572 10296 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:34.694608 10296 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:34.704607 10296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4af189bcdbfe412a9679d3fb79576932: Generated new TSK 0
I20260929 10:27:34.705350 10296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:34.718184 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10213
W20260929 10:27:34.723743 10198 connection.cc:535] client connection to 127.0.0.1:43493 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43493): Connection reset by peer (error 104)
2026-09-29T10:27:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:34 dist-test-slave-2rbl krb5kdc[10312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:34 dist-test-slave-2rbl krb5kdc[10312](info): set up 2 sockets
Sep 29 10:27:34 dist-test-slave-2rbl krb5kdc[10312](info): commencing operation
krb5kdc: starting...
W20260929 10:27:35.761953 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:35 dist-test-slave-2rbl krb5kdc[10312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:35.888623 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:36.018582 10325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:36.018891 10325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:36.018983 10325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:36.023000 10325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:36.023102 10325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:36.023140 10325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:36.023195 10325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:36.023233 10325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:36.028510 10325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:36.029592 10325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:36.030632 10325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:36.039261 10330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:36.039237 10331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:36.039309 10325 server_base.cc:1034] running on GCE node
W20260929 10:27:36.039255 10333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:36.039969 10325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:36.041059 10325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:36.042254 10325 hybrid_clock.cc:648] HybridClock initialized: now 1790677656042233 us; error 32 us; skew 500 ppm
Sep 29 10:27:36 dist-test-slave-2rbl krb5kdc[10312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:36.044951 10325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:36.045861 10325 webserver.cc:460] Webserver started at http://127.0.0.1:44893/ using document root <none> and password file <none>
I20260929 10:27:36.046417 10325 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:36.046509 10325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:36.046737 10325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:36.048748 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "bf4e866563bd48809437249e744959f6"
format_stamp: "Formatted at 2026-09-29 10:27:36 on dist-test-slave-2rbl"
server_key: "177ceba30f46bae8df374bd8ab3bbb94"
server_key_iv: "a616261f10e57f420ac32e6d2d107624"
server_key_version: "encryptionkey@0"
I20260929 10:27:36.049372 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "bf4e866563bd48809437249e744959f6"
format_stamp: "Formatted at 2026-09-29 10:27:36 on dist-test-slave-2rbl"
server_key: "177ceba30f46bae8df374bd8ab3bbb94"
server_key_iv: "a616261f10e57f420ac32e6d2d107624"
server_key_version: "encryptionkey@0"
I20260929 10:27:36.052966 10325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:27:36.055357 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:36.056569 10325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:36.056710 10325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "bf4e866563bd48809437249e744959f6"
format_stamp: "Formatted at 2026-09-29 10:27:36 on dist-test-slave-2rbl"
server_key: "177ceba30f46bae8df374bd8ab3bbb94"
server_key_iv: "a616261f10e57f420ac32e6d2d107624"
server_key_version: "encryptionkey@0"
I20260929 10:27:36.056820 10325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:36.061352 10325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:36.064500 10325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:36.064690 10325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:36.071722 10325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42995
I20260929 10:27:36.071762 10392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42995 every 8 connection(s)
I20260929 10:27:36.072664 10325 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:36.075053 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10325
I20260929 10:27:36.075191 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:36.075505 21638 external_mini_cluster.cc:1428] Setting key 3d56c189256c90c2f51d61f2811191be
I20260929 10:27:36.075637 10393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:36.080855 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:36.076908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:48659 (local address 127.0.0.1:42995)
0929 10:27:36.077380 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:36.077395 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:36.077440 (+ 45us) server_negotiation.cc:407] Connection header received
0929 10:27:36.078294 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:36.078312 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:36.078585 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:36.078941 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:36.079696 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:36.079701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:36.079722 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:36.079739 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:36.079791 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:36.079810 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:36.079959 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:36.080029 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 10:27:36.080188 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":123,"threads_started":1}
I20260929 10:27:36.082026 10393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6: Bootstrap starting.
I20260929 10:27:36.083580 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:36.084319 10393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:36.085109 10393 log.cc:824] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:36.087283 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:36.085606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:35521 (local address 127.0.0.1:42995)
0929 10:27:36.085745 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:36.085749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:36.085978 (+ 229us) server_negotiation.cc:407] Connection header received
0929 10:27:36.086112 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:36.086118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:36.086156 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:36.086232 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:36.086908 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:36.086912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:36.086914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:36.086926 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:36.086952 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:36.086964 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:36.087054 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:36.087137 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 10:27:36.087175 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:27:36.087386 10393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6: No bootstrap required, opened a new log
I20260929 10:27:36.090889 10393 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e866563bd48809437249e744959f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260929 10:27:36.091097 10393 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:36.091138 10393 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4e866563bd48809437249e744959f6, State: Initialized, Role: FOLLOWER
I20260929 10:27:36.091482 10393 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e866563bd48809437249e744959f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260929 10:27:36.091589 10393 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:36.091632 10393 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:36.091698 10393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:36.092494 10393 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e866563bd48809437249e744959f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260929 10:27:36.092916 10393 leader_election.cc:302] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4e866563bd48809437249e744959f6; no voters:
I20260929 10:27:36.093269 10393 leader_election.cc:288] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:36.093343 10403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:36.093570 10403 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [term 1 LEADER]: Becoming Leader. State: Replica: bf4e866563bd48809437249e744959f6, State: Running, Role: LEADER
I20260929 10:27:36.093878 10403 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e866563bd48809437249e744959f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } }
I20260929 10:27:36.094448 10393 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:36.095501 10405 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4e866563bd48809437249e744959f6. Latest consensus state: current_term: 1 leader_uuid: "bf4e866563bd48809437249e744959f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e866563bd48809437249e744959f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } } }
I20260929 10:27:36.095633 10405 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:36.095930 10404 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4e866563bd48809437249e744959f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e866563bd48809437249e744959f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42995 } } }
I20260929 10:27:36.096011 10412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:36.096029 10404 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:36.099016 10412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:36.103873 10412 catalog_manager.cc:1347] Generated new cluster ID: c20ed94836de421c95e4a1c698f3ae57
I20260929 10:27:36.103937 10412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:36.120718 10412 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:36.121081 10412 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:36.134753 10412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4e866563bd48809437249e744959f6: Generated new TSK 0
I20260929 10:27:36.135725 10412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:36.147040 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10325
2026-09-29T10:27:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:36 dist-test-slave-2rbl krb5kdc[10424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:36 dist-test-slave-2rbl krb5kdc[10424](info): set up 2 sockets
Sep 29 10:27:36 dist-test-slave-2rbl krb5kdc[10424](info): commencing operation
krb5kdc: starting...
W20260929 10:27:37.178001 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:37 dist-test-slave-2rbl krb5kdc[10424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:37.304121 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:37.412562 10437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:37.412897 10437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:37.412983 10437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:37.417276 10437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:37.417390 10437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:37.417433 10437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:37.417490 10437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:37.417527 10437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:37.422983 10437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:37.424099 10437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:37.425163 10437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:27:37.434003 10437 server_base.cc:1034] running on GCE node
W20260929 10:27:37.434068 10442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:37.434175 10445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:37.434068 10443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:37.435012 10437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:37.436209 10437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:37.437391 10437 hybrid_clock.cc:648] HybridClock initialized: now 1790677657437369 us; error 39 us; skew 500 ppm
Sep 29 10:27:37 dist-test-slave-2rbl krb5kdc[10424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:37.440204 10437 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:37.441115 10437 webserver.cc:460] Webserver started at http://127.0.0.1:46137/ using document root <none> and password file <none>
I20260929 10:27:37.441684 10437 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:37.441783 10437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:37.442008 10437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:37.444023 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "5deca4405a4c46fd86c617741d2bbcec"
format_stamp: "Formatted at 2026-09-29 10:27:37 on dist-test-slave-2rbl"
server_key: "9136c5297ae7118ceebffc18e0ff64e1"
server_key_iv: "b4fb5f86f6c50664bb06ed98f547660f"
server_key_version: "encryptionkey@0"
I20260929 10:27:37.444577 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "5deca4405a4c46fd86c617741d2bbcec"
format_stamp: "Formatted at 2026-09-29 10:27:37 on dist-test-slave-2rbl"
server_key: "9136c5297ae7118ceebffc18e0ff64e1"
server_key_iv: "b4fb5f86f6c50664bb06ed98f547660f"
server_key_version: "encryptionkey@0"
I20260929 10:27:37.448230 10437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:27:37.451143 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:37.452384 10437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 10:27:37.452523 10437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "5deca4405a4c46fd86c617741d2bbcec"
format_stamp: "Formatted at 2026-09-29 10:27:37 on dist-test-slave-2rbl"
server_key: "9136c5297ae7118ceebffc18e0ff64e1"
server_key_iv: "b4fb5f86f6c50664bb06ed98f547660f"
server_key_version: "encryptionkey@0"
I20260929 10:27:37.452651 10437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:37.463271 10437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:37.465996 10437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:37.466188 10437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:37.473043 10437 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38331
I20260929 10:27:37.473121 10504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38331 every 8 connection(s)
I20260929 10:27:37.474047 10437 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:37.476703 10505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:37.480104 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10437
I20260929 10:27:37.480280 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:37.480561 21638 external_mini_cluster.cc:1428] Setting key bb1cef0350cd3ba6c495d632cad54ecb
I20260929 10:27:37.482949 10505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec: Bootstrap starting.
I20260929 10:27:37.485652 10505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:37.486485 10505 log.cc:824] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:37.488559 10505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec: No bootstrap required, opened a new log
Sep 29 10:27:37 dist-test-slave-2rbl krb5kdc[10424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 10:27:37.491708 10505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5deca4405a4c46fd86c617741d2bbcec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260929 10:27:37.491966 10505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:37.492048 10505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deca4405a4c46fd86c617741d2bbcec, State: Initialized, Role: FOLLOWER
I20260929 10:27:37.492461 10505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5deca4405a4c46fd86c617741d2bbcec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260929 10:27:37.492614 10505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:37.492699 10505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:37.492828 10505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:37.493755 10505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5deca4405a4c46fd86c617741d2bbcec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260929 10:27:37.494087 10505 leader_election.cc:302] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5deca4405a4c46fd86c617741d2bbcec; no voters:
I20260929 10:27:37.494387 10505 leader_election.cc:288] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:37.494489 10510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:37.494709 10510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [term 1 LEADER]: Becoming Leader. State: Replica: 5deca4405a4c46fd86c617741d2bbcec, State: Running, Role: LEADER
I20260929 10:27:37.495078 10510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5deca4405a4c46fd86c617741d2bbcec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260929 10:27:37.495692 10505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:37.496338 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5deca4405a4c46fd86c617741d2bbcec. Latest consensus state: current_term: 1 leader_uuid: "5deca4405a4c46fd86c617741d2bbcec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5deca4405a4c46fd86c617741d2bbcec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260929 10:27:37.496513 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:37.496774 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5deca4405a4c46fd86c617741d2bbcec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5deca4405a4c46fd86c617741d2bbcec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260929 10:27:37.496881 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:37.496934 10518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:37.497005 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:37.481977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:49631 (local address 127.0.0.1:38331)
0929 10:27:37.482398 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:37.482417 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:37.482459 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:37.483260 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:37.483282 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:37.483710 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:37.484080 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:37.484733 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:37.486258 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:37.487127 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:37.487598 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:37.487713 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:37.490575 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:37.490580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:37.490592 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:37.490617 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:37.493059 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:37.493648 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:37.493654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:37.493659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:37.493774 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:37.494119 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:37.494124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:37.494130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:37.494844 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:37.495626 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:37.495792 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 10:27:37.496004 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260929 10:27:37.500630 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:37.501219 10518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:37.507148 10518 catalog_manager.cc:1347] Generated new cluster ID: f6f081602bfd4762bab8297fc844138a
I20260929 10:27:37.507212 10518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:37.511624 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:37.503156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:43541 (local address 127.0.0.1:38331)
0929 10:27:37.503281 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:37.503285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:37.503445 (+ 160us) server_negotiation.cc:407] Connection header received
0929 10:27:37.503580 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:37.503582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:37.503623 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:37.503715 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:37.504363 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:37.504913 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:37.505851 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:37.506208 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:37.506282 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:37.508184 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:37.508188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:37.508191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:37.508214 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:37.509834 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:37.510338 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:37.510342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:37.510343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:37.510417 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:37.510729 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:37.510732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:37.510733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:37.510927 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:37.511027 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:37.511393 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 10:27:37.511482 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 10:27:37.538105 10518 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:37.539441 10518 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:37.545317 10518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5deca4405a4c46fd86c617741d2bbcec: Generated new TSK 0
I20260929 10:27:37.546012 10518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:37.557103 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10437
W20260929 10:27:37.562400 10422 connection.cc:535] client connection to 127.0.0.1:38331 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38331): Connection reset by peer (error 104)
2026-09-29T10:27:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:37 dist-test-slave-2rbl krb5kdc[10536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:37 dist-test-slave-2rbl krb5kdc[10536](info): set up 2 sockets
Sep 29 10:27:37 dist-test-slave-2rbl krb5kdc[10536](info): commencing operation
krb5kdc: starting...
W20260929 10:27:38.594018 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:38 dist-test-slave-2rbl krb5kdc[10536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:38.721613 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:38.830021 10549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:38.830303 10549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:38.830389 10549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:38.834420 10549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:38.834489 10549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:38.834542 10549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:38.834582 10549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:38.834617 10549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:38.840085 10549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:38.841239 10549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:38.842312 10549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:38.850641 10554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:38.850651 10555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:38.850672 10557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:38.850862 10549 server_base.cc:1034] running on GCE node
I20260929 10:27:38.851621 10549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:38.852710 10549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:38.853878 10549 hybrid_clock.cc:648] HybridClock initialized: now 1790677658853859 us; error 30 us; skew 500 ppm
Sep 29 10:27:38 dist-test-slave-2rbl krb5kdc[10536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:38.856484 10549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:38.857453 10549 webserver.cc:460] Webserver started at http://127.0.0.1:34051/ using document root <none> and password file <none>
I20260929 10:27:38.858008 10549 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:38.858099 10549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:38.858327 10549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:38.860356 10549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "d8ff1afbb73f4ba780da811ce791a193"
format_stamp: "Formatted at 2026-09-29 10:27:38 on dist-test-slave-2rbl"
server_key: "a3a51533d9a9ac0cb7b0748913b13068"
server_key_iv: "7bdb4335fdc827cf6840091ee8d51325"
server_key_version: "encryptionkey@0"
I20260929 10:27:38.860924 10549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "d8ff1afbb73f4ba780da811ce791a193"
format_stamp: "Formatted at 2026-09-29 10:27:38 on dist-test-slave-2rbl"
server_key: "a3a51533d9a9ac0cb7b0748913b13068"
server_key_iv: "7bdb4335fdc827cf6840091ee8d51325"
server_key_version: "encryptionkey@0"
I20260929 10:27:38.864585 10549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 10:27:38.866993 10564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:38.868189 10549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 10:27:38.868341 10549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "d8ff1afbb73f4ba780da811ce791a193"
format_stamp: "Formatted at 2026-09-29 10:27:38 on dist-test-slave-2rbl"
server_key: "a3a51533d9a9ac0cb7b0748913b13068"
server_key_iv: "7bdb4335fdc827cf6840091ee8d51325"
server_key_version: "encryptionkey@0"
I20260929 10:27:38.868455 10549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:38.898365 10549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:38.901424 10549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:38.901636 10549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:38.908860 10549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40737
I20260929 10:27:38.908872 10616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40737 every 8 connection(s)
I20260929 10:27:38.909886 10549 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:38.912577 10617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:38.917907 10617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193: Bootstrap starting.
I20260929 10:27:38.918152 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10549
I20260929 10:27:38.918318 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:38.918591 21638 external_mini_cluster.cc:1428] Setting key 898f3f19f38386269d9a5ea3399b1a42
I20260929 10:27:38.920632 10617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:38.921389 10617 log.cc:824] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:38.923230 10617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193: No bootstrap required, opened a new log
I20260929 10:27:38.927043 10617 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff1afbb73f4ba780da811ce791a193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40737 } }
I20260929 10:27:38.927322 10617 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:38.927409 10617 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ff1afbb73f4ba780da811ce791a193, State: Initialized, Role: FOLLOWER
I20260929 10:27:38.927837 10617 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff1afbb73f4ba780da811ce791a193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40737 } }
I20260929 10:27:38.927906 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:38.919899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:44649 (local address 127.0.0.1:40737)
0929 10:27:38.920360 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:38.920375 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:38.920422 (+ 47us) server_negotiation.cc:407] Connection header received
0929 10:27:38.921114 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:38.921133 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:38.921590 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:38.921939 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:38.922353 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:38.923313 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:38.924297 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:38.924873 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:38.925059 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:38.926067 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:38.926072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:38.926085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:38.926472 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:38.926524 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:38.926547 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:38.926690 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:38.926874 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 10:27:38.927014 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260929 10:27:38.927995 10617 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:38.928082 10617 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:38.928188 10617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:38.929314 10617 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff1afbb73f4ba780da811ce791a193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40737 } }
I20260929 10:27:38.929654 10617 leader_election.cc:302] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8ff1afbb73f4ba780da811ce791a193; no voters:
I20260929 10:27:38.929984 10617 leader_election.cc:288] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:38.930073 10622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:38.930363 10622 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [term 1 LEADER]: Becoming Leader. State: Replica: d8ff1afbb73f4ba780da811ce791a193, State: Running, Role: LEADER
I20260929 10:27:38.930696 10622 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff1afbb73f4ba780da811ce791a193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40737 } }
I20260929 10:27:38.931098 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:38.931346 10617 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:38.932389 10623 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8ff1afbb73f4ba780da811ce791a193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff1afbb73f4ba780da811ce791a193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40737 } } }
I20260929 10:27:38.932509 10623 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:38.932835 10624 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8ff1afbb73f4ba780da811ce791a193. Latest consensus state: current_term: 1 leader_uuid: "d8ff1afbb73f4ba780da811ce791a193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ff1afbb73f4ba780da811ce791a193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40737 } } }
I20260929 10:27:38.932914 10624 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193 [sys.catalog]: This master's current role is: LEADER
W20260929 10:27:38.936563 10641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:38.936674 10641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:38.936846 10635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:38.937985 10635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:38.938982 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:38.933544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:60319 (local address 127.0.0.1:40737)
0929 10:27:38.933788 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:38.933794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:38.934218 (+ 424us) server_negotiation.cc:407] Connection header received
0929 10:27:38.934411 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:38.934414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:38.934468 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:38.934548 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:38.934903 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:38.935651 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:38.936928 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:38.937288 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:38.937408 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:38.938433 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:38.938437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:38.938440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:38.938454 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:38.938482 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:38.938496 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:38.938603 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:38.938806 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 10:27:38.938843 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260929 10:27:38.943903 10635 catalog_manager.cc:1347] Generated new cluster ID: c77ec519de5b443aa167a87178f679d4
I20260929 10:27:38.943979 10635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:38.957248 10635 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:38.958261 10635 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:38.963336 10635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8ff1afbb73f4ba780da811ce791a193: Generated new TSK 0
I20260929 10:27:38.964063 10635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:38.973824 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10549
W20260929 10:27:38.979342 10534 connection.cc:535] client connection to 127.0.0.1:40737 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40737): Connection reset by peer (error 104)
2026-09-29T10:27:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:39 dist-test-slave-2rbl krb5kdc[10648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:39 dist-test-slave-2rbl krb5kdc[10648](info): set up 2 sockets
Sep 29 10:27:39 dist-test-slave-2rbl krb5kdc[10648](info): commencing operation
krb5kdc: starting...
W20260929 10:27:40.022282 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:40 dist-test-slave-2rbl krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 10:27:40.144697 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:40.254061 10661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:40.254395 10661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:40.254500 10661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:40.258644 10661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:40.258780 10661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:40.258821 10661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:40.258858 10661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:40.258890 10661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:40.264437 10661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:40.265616 10661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:40.266669 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:40.275323 10669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:40.275357 10666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:40.275372 10667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:40.275496 10661 server_base.cc:1034] running on GCE node
I20260929 10:27:40.276180 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:40.277417 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:40.278587 10661 hybrid_clock.cc:648] HybridClock initialized: now 1790677660278573 us; error 29 us; skew 500 ppm
Sep 29 10:27:40 dist-test-slave-2rbl krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:40.285421 10661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 10:27:40.286311 10661 webserver.cc:460] Webserver started at http://127.0.0.1:40669/ using document root <none> and password file <none>
I20260929 10:27:40.286859 10661 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:40.286917 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:40.287094 10661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:40.289137 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "0050e1ad72824081a0254ac9666a4309"
format_stamp: "Formatted at 2026-09-29 10:27:40 on dist-test-slave-2rbl"
server_key: "5b0c09dc4c4f35cb2255a78683e4adf4"
server_key_iv: "39bd74c4c75764551bffe97f275dd7ed"
server_key_version: "encryptionkey@0"
I20260929 10:27:40.289736 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "0050e1ad72824081a0254ac9666a4309"
format_stamp: "Formatted at 2026-09-29 10:27:40 on dist-test-slave-2rbl"
server_key: "5b0c09dc4c4f35cb2255a78683e4adf4"
server_key_iv: "39bd74c4c75764551bffe97f275dd7ed"
server_key_version: "encryptionkey@0"
I20260929 10:27:40.293594 10661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 10:27:40.295972 10676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:40.297247 10661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:40.297387 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "0050e1ad72824081a0254ac9666a4309"
format_stamp: "Formatted at 2026-09-29 10:27:40 on dist-test-slave-2rbl"
server_key: "5b0c09dc4c4f35cb2255a78683e4adf4"
server_key_iv: "39bd74c4c75764551bffe97f275dd7ed"
server_key_version: "encryptionkey@0"
I20260929 10:27:40.297489 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:40.302153 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:40.305009 10661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:40.305217 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:40.312469 10661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34647
I20260929 10:27:40.312486 10728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34647 every 8 connection(s)
I20260929 10:27:40.313446 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:40.316200 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:40.320943 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10661
I20260929 10:27:40.321126 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:40.321437 21638 external_mini_cluster.cc:1428] Setting key 712623f666651fe1087f8daca9ce87de
I20260929 10:27:40.322189 10729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309: Bootstrap starting.
I20260929 10:27:40.324518 10729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:40.325510 10729 log.cc:824] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:40.326869 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:40.322910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:35439 (local address 127.0.0.1:34647)
0929 10:27:40.323366 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:40.323381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:40.323415 (+ 34us) server_negotiation.cc:407] Connection header received
0929 10:27:40.324031 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:40.324047 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:40.324355 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:40.324693 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:40.325519 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:40.325524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:40.325535 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:40.325544 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:40.325627 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:40.325664 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:40.325749 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:40.325917 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 10:27:40.326113 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":150,"threads_started":1}
I20260929 10:27:40.327574 10729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309: No bootstrap required, opened a new log
I20260929 10:27:40.329872 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:40.331681 10729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050e1ad72824081a0254ac9666a4309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260929 10:27:40.331960 10729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:40.332108 10729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0050e1ad72824081a0254ac9666a4309, State: Initialized, Role: FOLLOWER
I20260929 10:27:40.332597 10729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050e1ad72824081a0254ac9666a4309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260929 10:27:40.332744 10729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:40.332810 10729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:40.332891 10729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:40.333693 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:40.331899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:51685 (local address 127.0.0.1:34647)
0929 10:27:40.332062 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:40.332067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:40.332318 (+ 251us) server_negotiation.cc:407] Connection header received
0929 10:27:40.332450 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:40.332454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:40.332492 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:40.332574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:40.333310 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:40.333314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:40.333317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:40.333328 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:40.333355 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:40.333375 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:40.333463 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:40.333534 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 10:27:40.333578 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 10:27:40.333995 10729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050e1ad72824081a0254ac9666a4309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260929 10:27:40.334347 10729 leader_election.cc:302] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0050e1ad72824081a0254ac9666a4309; no voters:
I20260929 10:27:40.334653 10729 leader_election.cc:288] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:40.334735 10739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:40.334980 10739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [term 1 LEADER]: Becoming Leader. State: Replica: 0050e1ad72824081a0254ac9666a4309, State: Running, Role: LEADER
I20260929 10:27:40.335312 10739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050e1ad72824081a0254ac9666a4309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260929 10:27:40.335867 10729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:40.336906 10741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0050e1ad72824081a0254ac9666a4309. Latest consensus state: current_term: 1 leader_uuid: "0050e1ad72824081a0254ac9666a4309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050e1ad72824081a0254ac9666a4309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } } }
I20260929 10:27:40.337033 10741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:40.337430 10740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0050e1ad72824081a0254ac9666a4309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0050e1ad72824081a0254ac9666a4309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } } }
I20260929 10:27:40.337576 10740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:40.337463 10748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:40.340974 10748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:40.346947 10748 catalog_manager.cc:1347] Generated new cluster ID: 4e32d4d919d34fb9ac309a57c8dd5843
I20260929 10:27:40.347034 10748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:40.374579 10748 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:40.374940 10748 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:40.384670 10748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0050e1ad72824081a0254ac9666a4309: Generated new TSK 0
I20260929 10:27:40.385545 10748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:40.397990 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10661
2026-09-29T10:27:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:40 dist-test-slave-2rbl krb5kdc[10760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:40 dist-test-slave-2rbl krb5kdc[10760](info): set up 2 sockets
Sep 29 10:27:40 dist-test-slave-2rbl krb5kdc[10760](info): commencing operation
krb5kdc: starting...
W20260929 10:27:41.434022 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:41 dist-test-slave-2rbl krb5kdc[10760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:41.560449 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:37669
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:37669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:41.676738 10773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:41.677053 10773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:41.677145 10773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:41.681313 10773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:41.681393 10773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:41.681416 10773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:41.681437 10773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:41.681454 10773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:41.686959 10773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:37669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:37669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:41.688036 10773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:41.689043 10773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:27:41.697619 10773 server_base.cc:1034] running on GCE node
W20260929 10:27:41.697604 10778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:41.697666 10781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:41.697876 10779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:41.698554 10773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:41.699672 10773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:41.700906 10773 hybrid_clock.cc:648] HybridClock initialized: now 1790677661700834 us; error 54 us; skew 500 ppm
Sep 29 10:27:41 dist-test-slave-2rbl krb5kdc[10760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677661, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:41.703747 10773 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:41.704680 10773 webserver.cc:460] Webserver started at http://127.21.33.190:45453/ using document root <none> and password file <none>
I20260929 10:27:41.705282 10773 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:41.705355 10773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:41.705549 10773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:41.707612 10773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "db029af626f845fdb072425d1143ba4e"
format_stamp: "Formatted at 2026-09-29 10:27:41 on dist-test-slave-2rbl"
server_key: "bdf8a0ca2b47c4286402f274f8fca2b5"
server_key_iv: "a51dfd294a810ceaf4ea362c605e0f35"
server_key_version: "encryptionkey@0"
I20260929 10:27:41.708173 10773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "db029af626f845fdb072425d1143ba4e"
format_stamp: "Formatted at 2026-09-29 10:27:41 on dist-test-slave-2rbl"
server_key: "bdf8a0ca2b47c4286402f274f8fca2b5"
server_key_iv: "a51dfd294a810ceaf4ea362c605e0f35"
server_key_version: "encryptionkey@0"
I20260929 10:27:41.712173 10773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 10:27:41.714720 10788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:41.715992 10773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:27:41.716104 10773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "db029af626f845fdb072425d1143ba4e"
format_stamp: "Formatted at 2026-09-29 10:27:41 on dist-test-slave-2rbl"
server_key: "bdf8a0ca2b47c4286402f274f8fca2b5"
server_key_iv: "a51dfd294a810ceaf4ea362c605e0f35"
server_key_version: "encryptionkey@0"
I20260929 10:27:41.716197 10773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:41.738595 10773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:41.741918 10773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:41.742169 10773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:41.749830 10773 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:37669
I20260929 10:27:41.749850 10840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:37669 every 8 connection(s)
I20260929 10:27:41.750875 10773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:41.753788 10841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:41.757143 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10773
I20260929 10:27:41.757314 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:41.757596 21638 external_mini_cluster.cc:1428] Setting key 97d28ae0016dee024e28d85ed2d6889f
I20260929 10:27:41.759940 10841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e: Bootstrap starting.
I20260929 10:27:41.762563 10841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:41.763460 10841 log.cc:824] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:41.765771 10841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e: No bootstrap required, opened a new log
Sep 29 10:27:41 dist-test-slave-2rbl krb5kdc[10760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:41.769974 10841 raft_consensus.cc:348] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db029af626f845fdb072425d1143ba4e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37669 } }
I20260929 10:27:41.770268 10841 raft_consensus.cc:374] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:41.770339 10841 raft_consensus.cc:729] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db029af626f845fdb072425d1143ba4e, State: Initialized, Role: FOLLOWER
I20260929 10:27:41.770828 10841 consensus_queue.cc:260] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db029af626f845fdb072425d1143ba4e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37669 } }
I20260929 10:27:41.770989 10841 raft_consensus.cc:388] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:41.771063 10841 raft_consensus.cc:482] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:41.771185 10841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:41.772286 10841 raft_consensus.cc:504] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db029af626f845fdb072425d1143ba4e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37669 } }
I20260929 10:27:41.772670 10841 leader_election.cc:302] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db029af626f845fdb072425d1143ba4e; no voters:
I20260929 10:27:41.773016 10841 leader_election.cc:288] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:41.773131 10846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:41.773536 10846 raft_consensus.cc:686] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [term 1 LEADER]: Becoming Leader. State: Replica: db029af626f845fdb072425d1143ba4e, State: Running, Role: LEADER
I20260929 10:27:41.773851 10846 consensus_queue.cc:237] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db029af626f845fdb072425d1143ba4e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37669 } }
I20260929 10:27:41.774096 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:41.758986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58235 (local address 127.21.33.190:37669)
0929 10:27:41.759396 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:41.759412 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:41.759448 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:27:41.760142 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:41.760162 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:41.760512 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:41.760801 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:41.761768 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:41.762871 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:41.764032 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:41.764592 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:41.764712 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:41.767696 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:41.767704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:41.767730 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:41.767763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:41.770515 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:41.771054 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:41.771062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:41.771068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:41.771238 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:41.771631 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:41.771633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:41.771634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:41.772010 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:41.772178 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:41.772602 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 10:27:41.772873 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260929 10:27:41.774421 10841 sys_catalog.cc:564] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:41.775087 10846 sys_catalog.cc:455] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader db029af626f845fdb072425d1143ba4e. Latest consensus state: current_term: 1 leader_uuid: "db029af626f845fdb072425d1143ba4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db029af626f845fdb072425d1143ba4e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37669 } } }
I20260929 10:27:41.775813 10846 sys_catalog.cc:458] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:41.775699 10847 sys_catalog.cc:455] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db029af626f845fdb072425d1143ba4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db029af626f845fdb072425d1143ba4e" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 37669 } } }
I20260929 10:27:41.776111 10847 sys_catalog.cc:458] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:41.777563 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 10:27:41.779464 10861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:41.779557 10861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:41.779641 10856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:41.780507 10856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:41.786800 10856 catalog_manager.cc:1347] Generated new cluster ID: 47d0e2eaec8d408d909d51946dd0508b
I20260929 10:27:41.786859 10856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:41.788755 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:41.779687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57833 (local address 127.21.33.190:37669)
0929 10:27:41.779791 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:41.779795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:41.780308 (+ 513us) server_negotiation.cc:407] Connection header received
0929 10:27:41.780416 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:41.780419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:41.780464 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:41.780549 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:41.781458 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:41.782204 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:41.783138 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:41.783407 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:41.783520 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:41.785111 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:41.785115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:41.785118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:41.785142 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:41.786887 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:41.787484 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:41.787488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:41.787490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:41.787583 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:41.787921 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:41.787924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:41.787926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:41.788116 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:41.788248 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:41.788523 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 10:27:41.788612 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:27:41.803630 10856 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:41.804864 10856 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:41.816644 10856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db029af626f845fdb072425d1143ba4e: Generated new TSK 0
I20260929 10:27:41.817575 10856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:41.827232 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10773
W20260929 10:27:41.832644 10758 connection.cc:535] client connection to 127.21.33.190:37669 recv error: Network error: failed to read from TLS socket (remote: 127.21.33.190:37669): Connection reset by peer (error 104)
2026-09-29T10:27:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:41 dist-test-slave-2rbl krb5kdc[10872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:41 dist-test-slave-2rbl krb5kdc[10872](info): set up 2 sockets
Sep 29 10:27:41 dist-test-slave-2rbl krb5kdc[10872](info): commencing operation
krb5kdc: starting...
W20260929 10:27:42.874022 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:42 dist-test-slave-2rbl krb5kdc[10872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:43.000530 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:42307
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:43.110966 10885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:43.111222 10885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:43.111263 10885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:43.115319 10885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:43.115375 10885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:43.115394 10885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:43.115445 10885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:43.115475 10885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:43.120664 10885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:42307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:43.121799 10885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:43.122822 10885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:43.131084 10890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:43.131143 10893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:43.131117 10891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:43.131294 10885 server_base.cc:1034] running on GCE node
I20260929 10:27:43.132022 10885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:43.133315 10885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:43.134505 10885 hybrid_clock.cc:648] HybridClock initialized: now 1790677663134489 us; error 29 us; skew 500 ppm
Sep 29 10:27:43 dist-test-slave-2rbl krb5kdc[10872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677663, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:43.137389 10885 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:43.138279 10885 webserver.cc:460] Webserver started at http://127.21.33.190:36665/ using document root <none> and password file <none>
I20260929 10:27:43.138865 10885 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:43.138962 10885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:43.139194 10885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:43.141311 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5"
format_stamp: "Formatted at 2026-09-29 10:27:43 on dist-test-slave-2rbl"
server_key: "96d58ce63ebe5a69c7a8db8be3bd2beb"
server_key_iv: "f95ef56a07f33a06d317fd46deac44fa"
server_key_version: "encryptionkey@0"
I20260929 10:27:43.141918 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5"
format_stamp: "Formatted at 2026-09-29 10:27:43 on dist-test-slave-2rbl"
server_key: "96d58ce63ebe5a69c7a8db8be3bd2beb"
server_key_iv: "f95ef56a07f33a06d317fd46deac44fa"
server_key_version: "encryptionkey@0"
I20260929 10:27:43.145809 10885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 10:27:43.148343 10900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:43.149644 10885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:43.149788 10885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5"
format_stamp: "Formatted at 2026-09-29 10:27:43 on dist-test-slave-2rbl"
server_key: "96d58ce63ebe5a69c7a8db8be3bd2beb"
server_key_iv: "f95ef56a07f33a06d317fd46deac44fa"
server_key_version: "encryptionkey@0"
I20260929 10:27:43.149901 10885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:43.162407 10885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:43.166100 10885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:43.166328 10885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:43.174103 10885 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:42307
I20260929 10:27:43.174161 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:42307 every 8 connection(s)
I20260929 10:27:43.175109 10885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:43.176689 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10885
I20260929 10:27:43.176837 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:43.177160 21638 external_mini_cluster.cc:1428] Setting key bcffa6cc14947043ed82f1a1c99701c1
I20260929 10:27:43.178623 10953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:43.184322 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5: Bootstrap starting.
I20260929 10:27:43.186232 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:43.178577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36659 (local address 127.21.33.190:42307)
0929 10:27:43.179045 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:43.179060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:43.179097 (+ 37us) server_negotiation.cc:407] Connection header received
0929 10:27:43.179749 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:43.179766 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:43.180117 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:43.180461 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:43.180861 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:43.181910 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:43.182813 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:43.183295 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:43.183490 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:43.184455 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:43.184462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:43.184479 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:43.184886 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:43.184941 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:43.184964 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:43.185112 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:43.185342 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 10:27:43.185488 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":163,"threads_started":1}
I20260929 10:27:43.186817 10953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:43.187551 10953 log.cc:824] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:43.189256 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:43.189775 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5: No bootstrap required, opened a new log
I20260929 10:27:43.194389 10953 raft_consensus.cc:348] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42307 } }
I20260929 10:27:43.194690 10953 raft_consensus.cc:374] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:43.194769 10953 raft_consensus.cc:729] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b371bb59da494e25b8f1a7ebcfdbf9a5, State: Initialized, Role: FOLLOWER
I20260929 10:27:43.195294 10953 consensus_queue.cc:260] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42307 } }
I20260929 10:27:43.195472 10953 raft_consensus.cc:388] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:43.195580 10953 raft_consensus.cc:482] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:43.195703 10953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:43.195744 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:43.191302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38771 (local address 127.21.33.190:42307)
0929 10:27:43.191449 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:43.191453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:43.191701 (+ 248us) server_negotiation.cc:407] Connection header received
0929 10:27:43.191807 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:43.191811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:43.191868 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:43.191962 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:43.192225 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:43.193034 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:43.193903 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:43.194237 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:43.194346 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:43.195180 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:43.195183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:43.195185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:43.195195 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:43.195218 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:43.195228 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:43.195297 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:43.195535 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 10:27:43.195573 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 10:27:43.196724 10953 raft_consensus.cc:504] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42307 } }
I20260929 10:27:43.197080 10953 leader_election.cc:302] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b371bb59da494e25b8f1a7ebcfdbf9a5; no voters:
I20260929 10:27:43.197417 10953 leader_election.cc:288] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:43.197533 10963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:43.197786 10963 raft_consensus.cc:686] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [term 1 LEADER]: Becoming Leader. State: Replica: b371bb59da494e25b8f1a7ebcfdbf9a5, State: Running, Role: LEADER
I20260929 10:27:43.198207 10963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42307 } }
I20260929 10:27:43.198683 10953 sys_catalog.cc:564] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:43.199888 10965 sys_catalog.cc:455] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b371bb59da494e25b8f1a7ebcfdbf9a5. Latest consensus state: current_term: 1 leader_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42307 } } }
I20260929 10:27:43.200018 10965 sys_catalog.cc:458] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:43.200320 10964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b371bb59da494e25b8f1a7ebcfdbf9a5" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 42307 } } }
I20260929 10:27:43.200407 10972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:43.200433 10964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:43.203728 10972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:43.209112 10972 catalog_manager.cc:1347] Generated new cluster ID: bf8b0ca9295d473ea041a1aca4f5dbc6
I20260929 10:27:43.209187 10972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:43.238588 10972 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:43.239619 10972 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:43.248072 10972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b371bb59da494e25b8f1a7ebcfdbf9a5: Generated new TSK 0
I20260929 10:27:43.248862 10972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:43.263674 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10885
W20260929 10:27:43.268765 10870 connection.cc:535] client connection to 127.21.33.190:42307 recv error: Network error: failed to read from TLS socket (remote: 127.21.33.190:42307): Connection reset by peer (error 104)
2026-09-29T10:27:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:43 dist-test-slave-2rbl krb5kdc[10984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:43 dist-test-slave-2rbl krb5kdc[10984](info): set up 2 sockets
Sep 29 10:27:43 dist-test-slave-2rbl krb5kdc[10984](info): commencing operation
krb5kdc: starting...
W20260929 10:27:44.298056 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:44 dist-test-slave-2rbl krb5kdc[10984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:44.422467 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:43001
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:38081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:44.534119 10997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:44.534492 10997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:44.534560 10997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:44.539825 10997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:44.539891 10997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:44.539911 10997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:44.539929 10997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:44.539944 10997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:44.545310 10997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:38081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:43001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:44.546350 10997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:44.547324 10997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:44.555698 11005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:44.555735 11003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:44.555789 10997 server_base.cc:1034] running on GCE node
W20260929 10:27:44.555711 11002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:44.556499 10997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:44.557663 10997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:44.558866 10997 hybrid_clock.cc:648] HybridClock initialized: now 1790677664558837 us; error 38 us; skew 500 ppm
Sep 29 10:27:44 dist-test-slave-2rbl krb5kdc[10984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677664, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:44.561676 10997 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:44.562549 10997 webserver.cc:460] Webserver started at http://127.21.33.190:42511/ using document root <none> and password file <none>
I20260929 10:27:44.563129 10997 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:44.563220 10997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:44.563445 10997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:44.565526 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "0c8ea6181f114ff4b791aa0dcf57e84d"
format_stamp: "Formatted at 2026-09-29 10:27:44 on dist-test-slave-2rbl"
server_key: "35225fd04fac6b84fca3dabe4c0048fd"
server_key_iv: "fb54e88c25e350cd63f23d6ff164a4a7"
server_key_version: "encryptionkey@0"
I20260929 10:27:44.566094 10997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "0c8ea6181f114ff4b791aa0dcf57e84d"
format_stamp: "Formatted at 2026-09-29 10:27:44 on dist-test-slave-2rbl"
server_key: "35225fd04fac6b84fca3dabe4c0048fd"
server_key_iv: "fb54e88c25e350cd63f23d6ff164a4a7"
server_key_version: "encryptionkey@0"
I20260929 10:27:44.570091 10997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:27:44.572587 11012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:44.573858 10997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:44.574004 10997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "0c8ea6181f114ff4b791aa0dcf57e84d"
format_stamp: "Formatted at 2026-09-29 10:27:44 on dist-test-slave-2rbl"
server_key: "35225fd04fac6b84fca3dabe4c0048fd"
server_key_iv: "fb54e88c25e350cd63f23d6ff164a4a7"
server_key_version: "encryptionkey@0"
I20260929 10:27:44.574116 10997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:44.578552 10997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:44.581367 10997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:44.581568 10997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:44.588696 10997 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:43001
I20260929 10:27:44.588734 11064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:43001 every 8 connection(s)
I20260929 10:27:44.589648 10997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:44.592278 11065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:44.597827 11065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d: Bootstrap starting.
I20260929 10:27:44.598829 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 10997
I20260929 10:27:44.598969 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:44.599257 21638 external_mini_cluster.cc:1428] Setting key 1f0875fa658641aed689f094662a62d7
I20260929 10:27:44.600591 11065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:44.601485 11065 log.cc:824] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:44.603693 11065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d: No bootstrap required, opened a new log
I20260929 10:27:44.604676 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:44.600372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40239 (local address 127.21.33.190:43001)
0929 10:27:44.600903 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:44.600921 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:44.600963 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:44.601874 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:44.601907 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:44.602225 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:44.602583 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:44.603308 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:44.603314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:44.603328 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:44.603339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:44.603414 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:44.603441 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:44.603556 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:44.603623 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 10:27:44.603774 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":144,"threads_started":1}
I20260929 10:27:44.607466 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:44.608050 11065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43001 } }
I20260929 10:27:44.608263 11065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:44.608330 11065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8ea6181f114ff4b791aa0dcf57e84d, State: Initialized, Role: FOLLOWER
I20260929 10:27:44.608839 11065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43001 } }
I20260929 10:27:44.608994 11065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:44.609059 11065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:44.609154 11065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:44.610283 11065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43001 } }
I20260929 10:27:44.610670 11065 leader_election.cc:302] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8ea6181f114ff4b791aa0dcf57e84d; no voters:
I20260929 10:27:44.611003 11065 leader_election.cc:288] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:44.611096 11075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:44.611267 11075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8ea6181f114ff4b791aa0dcf57e84d, State: Running, Role: LEADER
I20260929 10:27:44.611459 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:44.609677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60899 (local address 127.21.33.190:43001)
0929 10:27:44.609806 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:44.609811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:44.609964 (+ 153us) server_negotiation.cc:407] Connection header received
0929 10:27:44.610113 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:44.610116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:44.610149 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:44.610244 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:44.611009 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:44.611013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:44.611015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:44.611027 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:44.611054 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:44.611069 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:44.611145 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:44.611275 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 10:27:44.611315 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 10:27:44.611743 11075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43001 } }
I20260929 10:27:44.612331 11065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:44.613561 11076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8ea6181f114ff4b791aa0dcf57e84d. Latest consensus state: current_term: 1 leader_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43001 } } }
I20260929 10:27:44.613687 11076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:44.613951 11077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8ea6181f114ff4b791aa0dcf57e84d" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 43001 } } }
I20260929 10:27:44.614051 11084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:44.614050 11077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:44.617010 11084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:44.622448 11084 catalog_manager.cc:1347] Generated new cluster ID: d0df39b3775e4577a26c4d23f057a499
I20260929 10:27:44.622514 11084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:44.636670 11084 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:44.637020 11084 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:44.647132 11084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c8ea6181f114ff4b791aa0dcf57e84d: Generated new TSK 0
I20260929 10:27:44.647876 11084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:44.652937 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 10997
2026-09-29T10:27:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:44 dist-test-slave-2rbl krb5kdc[11096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:44 dist-test-slave-2rbl krb5kdc[11096](info): set up 2 sockets
Sep 29 10:27:44 dist-test-slave-2rbl krb5kdc[11096](info): commencing operation
krb5kdc: starting...
W20260929 10:27:45.690037 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:45 dist-test-slave-2rbl krb5kdc[11096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:45.811978 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39221
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:39221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:45.921026 11109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:45.921375 11109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:45.921456 11109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:45.925608 11109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:45.925709 11109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:45.925748 11109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:45.925800 11109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:45.925838 11109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:45.931288 11109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:39221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:45.932404 11109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:45.933468 11109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:45.941865 11114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:45.941895 11117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:45.941960 11109 server_base.cc:1034] running on GCE node
W20260929 10:27:45.941865 11115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:45.942759 11109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:45.943775 11109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:45.944939 11109 hybrid_clock.cc:648] HybridClock initialized: now 1790677665944920 us; error 31 us; skew 500 ppm
Sep 29 10:27:45 dist-test-slave-2rbl krb5kdc[11096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677665, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:45.956614 11109 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:45.957813 11109 webserver.cc:460] Webserver started at http://127.21.33.190:46429/ using document root <none> and password file <none>
I20260929 10:27:45.958390 11109 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:45.958455 11109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:45.958642 11109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:45.960999 11109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "98a740c03464455cbb5dde1717009f1b"
format_stamp: "Formatted at 2026-09-29 10:27:45 on dist-test-slave-2rbl"
server_key: "7df43512b4ed81927b03049367e4735a"
server_key_iv: "32346bfbfea54d6ff78fe916b51274e6"
server_key_version: "encryptionkey@0"
I20260929 10:27:45.961635 11109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "98a740c03464455cbb5dde1717009f1b"
format_stamp: "Formatted at 2026-09-29 10:27:45 on dist-test-slave-2rbl"
server_key: "7df43512b4ed81927b03049367e4735a"
server_key_iv: "32346bfbfea54d6ff78fe916b51274e6"
server_key_version: "encryptionkey@0"
I20260929 10:27:45.965442 11109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:27:45.967942 11124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:45.969496 11109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:27:45.969623 11109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "98a740c03464455cbb5dde1717009f1b"
format_stamp: "Formatted at 2026-09-29 10:27:45 on dist-test-slave-2rbl"
server_key: "7df43512b4ed81927b03049367e4735a"
server_key_iv: "32346bfbfea54d6ff78fe916b51274e6"
server_key_version: "encryptionkey@0"
I20260929 10:27:45.969719 11109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:45.999341 11109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:46.003156 11109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:46.003373 11109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:46.011618 11109 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:39221
I20260929 10:27:46.011919 11176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:39221 every 8 connection(s)
I20260929 10:27:46.012663 11109 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:46.016541 11177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:46.018594 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11109
I20260929 10:27:46.018769 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:46.019091 21638 external_mini_cluster.cc:1428] Setting key 57de1f389ec7abb851292eb94dce5970
I20260929 10:27:46.024178 11177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b: Bootstrap starting.
I20260929 10:27:46.026481 11177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:46.027204 11177 log.cc:824] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b: Log is configured to *not* fsync() on all Append() calls
Sep 29 10:27:46 dist-test-slave-2rbl krb5kdc[11096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:46.029297 11177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b: No bootstrap required, opened a new log
I20260929 10:27:46.032855 11177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a740c03464455cbb5dde1717009f1b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39221 } }
I20260929 10:27:46.033093 11177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:46.033176 11177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a740c03464455cbb5dde1717009f1b, State: Initialized, Role: FOLLOWER
I20260929 10:27:46.033650 11177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a740c03464455cbb5dde1717009f1b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39221 } }
I20260929 10:27:46.033772 11177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:46.033883 11177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:46.033885 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:46.020558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38643 (local address 127.21.33.190:39221)
0929 10:27:46.021038 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:46.021058 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:46.021100 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:46.021951 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:46.021975 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:46.022402 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:46.022776 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:46.023545 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:46.024590 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:46.025523 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:46.025993 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:46.026176 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:46.028983 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:46.028989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:46.029001 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:46.029026 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:46.030740 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:46.031172 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:46.031178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:46.031190 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:46.031297 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:46.031598 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:46.031601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:46.031602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:46.032037 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:46.032257 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:46.032544 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 10:27:46.032831 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
I20260929 10:27:46.034005 11177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:46.034897 11177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a740c03464455cbb5dde1717009f1b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39221 } }
I20260929 10:27:46.035255 11177 leader_election.cc:302] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a740c03464455cbb5dde1717009f1b; no voters:
I20260929 10:27:46.035567 11177 leader_election.cc:288] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:46.035712 11182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:46.035983 11182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [term 1 LEADER]: Becoming Leader. State: Replica: 98a740c03464455cbb5dde1717009f1b, State: Running, Role: LEADER
I20260929 10:27:46.036334 11182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a740c03464455cbb5dde1717009f1b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39221 } }
I20260929 10:27:46.036824 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:46.036923 11177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:46.037997 11184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98a740c03464455cbb5dde1717009f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a740c03464455cbb5dde1717009f1b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39221 } } }
I20260929 10:27:46.038118 11184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:46.038902 11183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a740c03464455cbb5dde1717009f1b. Latest consensus state: current_term: 1 leader_uuid: "98a740c03464455cbb5dde1717009f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a740c03464455cbb5dde1717009f1b" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39221 } } }
I20260929 10:27:46.039001 11183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b [sys.catalog]: This master's current role is: LEADER
W20260929 10:27:46.042133 11202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:46.042225 11202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:46.042320 11194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:46.042963 11194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:46.048590 11194 catalog_manager.cc:1347] Generated new cluster ID: 387e2a78128e489f93aefd55f7045d2c
I20260929 10:27:46.048666 11194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:46.048714 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:46.039466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57739 (local address 127.21.33.190:39221)
0929 10:27:46.039581 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:46.039586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:46.040043 (+ 457us) server_negotiation.cc:407] Connection header received
0929 10:27:46.040160 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:46.040163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:46.040208 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:46.040291 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:46.040970 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:46.041816 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:46.042920 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:46.043289 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:46.043368 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:46.045176 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:46.045180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:46.045183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:46.045248 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:46.046850 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:46.047361 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:46.047364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:46.047366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:46.047457 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:46.047764 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:46.047768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:46.047770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:46.047990 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:46.048125 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:46.048397 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 10:27:46.048520 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 10:27:46.069178 11194 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:46.070494 11194 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:46.077339 11194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98a740c03464455cbb5dde1717009f1b: Generated new TSK 0
I20260929 10:27:46.078047 11194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:46.087972 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11109
2026-09-29T10:27:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:46 dist-test-slave-2rbl krb5kdc[11208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:46 dist-test-slave-2rbl krb5kdc[11208](info): set up 2 sockets
Sep 29 10:27:46 dist-test-slave-2rbl krb5kdc[11208](info): commencing operation
krb5kdc: starting...
W20260929 10:27:47.122121 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:47 dist-test-slave-2rbl krb5kdc[11208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:47.253732 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:41793
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:41793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:47.372833 11221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:47.373201 11221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:47.373313 11221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:47.377493 11221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:47.377584 11221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:47.377624 11221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:47.377660 11221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:47.377692 11221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:47.383246 11221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:41793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:47.384429 11221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:47.385576 11221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:47.394760 11229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:47.394682 11227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:47.394740 11221 server_base.cc:1034] running on GCE node
W20260929 10:27:47.394691 11226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:47.395550 11221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:47.396683 11221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:47.397850 11221 hybrid_clock.cc:648] HybridClock initialized: now 1790677667397835 us; error 28 us; skew 500 ppm
Sep 29 10:27:47 dist-test-slave-2rbl krb5kdc[11208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677667, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:47.400640 11221 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:47.401682 11221 webserver.cc:460] Webserver started at http://127.21.33.190:38959/ using document root <none> and password file <none>
I20260929 10:27:47.402287 11221 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:47.402383 11221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:47.402624 11221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:47.404747 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "e83473fbd48d42fa9fe19662e4bb614f"
format_stamp: "Formatted at 2026-09-29 10:27:47 on dist-test-slave-2rbl"
server_key: "56bb77343da159b04ca1f8ae739694de"
server_key_iv: "b61c3fed7497fc491878a930fb6e52d0"
server_key_version: "encryptionkey@0"
I20260929 10:27:47.405393 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "e83473fbd48d42fa9fe19662e4bb614f"
format_stamp: "Formatted at 2026-09-29 10:27:47 on dist-test-slave-2rbl"
server_key: "56bb77343da159b04ca1f8ae739694de"
server_key_iv: "b61c3fed7497fc491878a930fb6e52d0"
server_key_version: "encryptionkey@0"
I20260929 10:27:47.409250 11221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 10:27:47.411751 11236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:47.412997 11221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:47.413147 11221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "e83473fbd48d42fa9fe19662e4bb614f"
format_stamp: "Formatted at 2026-09-29 10:27:47 on dist-test-slave-2rbl"
server_key: "56bb77343da159b04ca1f8ae739694de"
server_key_iv: "b61c3fed7497fc491878a930fb6e52d0"
server_key_version: "encryptionkey@0"
I20260929 10:27:47.413344 11221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:47.424649 11221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:47.427856 11221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:47.428069 11221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:47.435360 11221 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:41793
I20260929 10:27:47.435384 11288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:41793 every 8 connection(s)
I20260929 10:27:47.436342 11221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:47.439023 11289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:47.440526 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11221
I20260929 10:27:47.440660 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:47.440977 21638 external_mini_cluster.cc:1428] Setting key 7c915d1e178b739a668bd28459bcbef4
I20260929 10:27:47.445520 11289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f: Bootstrap starting.
I20260929 10:27:47.447751 11289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:47.448519 11289 log.cc:824] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:47.450230 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:47.442459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33175 (local address 127.21.33.190:41793)
0929 10:27:47.442883 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:47.442897 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:47.442933 (+ 36us) server_negotiation.cc:407] Connection header received
0929 10:27:47.443576 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:47.443593 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:47.443941 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:47.444280 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:47.444683 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:47.445843 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:47.446757 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:47.447235 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:47.447407 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:47.448274 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:47.448285 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:47.448299 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:47.448729 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:47.448786 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:47.448808 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:47.448917 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:47.449062 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 10:27:47.449323 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260929 10:27:47.450750 11289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f: No bootstrap required, opened a new log
I20260929 10:27:47.453138 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:47.454819 11289 raft_consensus.cc:348] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83473fbd48d42fa9fe19662e4bb614f" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41793 } }
I20260929 10:27:47.455128 11289 raft_consensus.cc:374] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:47.455238 11289 raft_consensus.cc:729] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83473fbd48d42fa9fe19662e4bb614f, State: Initialized, Role: FOLLOWER
I20260929 10:27:47.455839 11289 consensus_queue.cc:260] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83473fbd48d42fa9fe19662e4bb614f" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41793 } }
I20260929 10:27:47.456028 11289 raft_consensus.cc:388] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:47.456103 11289 raft_consensus.cc:482] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:47.456208 11289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:47.457383 11289 raft_consensus.cc:504] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83473fbd48d42fa9fe19662e4bb614f" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41793 } }
I20260929 10:27:47.457723 11289 leader_election.cc:302] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e83473fbd48d42fa9fe19662e4bb614f; no voters:
I20260929 10:27:47.458053 11289 leader_election.cc:288] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:47.458164 11299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:47.458407 11299 raft_consensus.cc:686] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [term 1 LEADER]: Becoming Leader. State: Replica: e83473fbd48d42fa9fe19662e4bb614f, State: Running, Role: LEADER
I20260929 10:27:47.459039 11299 consensus_queue.cc:237] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83473fbd48d42fa9fe19662e4bb614f" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41793 } }
I20260929 10:27:47.459440 11289 sys_catalog.cc:564] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:47.459842 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:47.455272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45193 (local address 127.21.33.190:41793)
0929 10:27:47.455406 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:47.455410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:47.455609 (+ 199us) server_negotiation.cc:407] Connection header received
0929 10:27:47.455727 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:47.455730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:47.455778 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:47.455860 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:47.456149 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:47.456889 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:47.457801 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:47.458218 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:47.458326 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:47.459241 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:47.459244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:47.459247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:47.459259 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:47.459286 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:47.459298 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:47.459358 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:47.459661 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 10:27:47.459713 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 10:27:47.460686 11301 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e83473fbd48d42fa9fe19662e4bb614f. Latest consensus state: current_term: 1 leader_uuid: "e83473fbd48d42fa9fe19662e4bb614f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83473fbd48d42fa9fe19662e4bb614f" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41793 } } }
I20260929 10:27:47.460814 11301 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:47.461146 11300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e83473fbd48d42fa9fe19662e4bb614f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83473fbd48d42fa9fe19662e4bb614f" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41793 } } }
I20260929 10:27:47.461274 11300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:47.461638 11308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:47.464488 11308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:47.469980 11308 catalog_manager.cc:1347] Generated new cluster ID: 02d6c500d04b4254a104e4268dbcf0f7
I20260929 10:27:47.470045 11308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:47.491195 11308 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:47.492164 11308 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:47.495630 11308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e83473fbd48d42fa9fe19662e4bb614f: Generated new TSK 0
I20260929 10:27:47.496320 11308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:47.502555 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11221
2026-09-29T10:27:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:47 dist-test-slave-2rbl krb5kdc[11320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:47 dist-test-slave-2rbl krb5kdc[11320](info): set up 2 sockets
Sep 29 10:27:47 dist-test-slave-2rbl krb5kdc[11320](info): commencing operation
krb5kdc: starting...
W20260929 10:27:48.546029 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:48 dist-test-slave-2rbl krb5kdc[11320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:48.670639 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39793
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:43655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:39793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:48.782194 11333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:48.782495 11333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:48.782575 11333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:48.786682 11333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:48.786774 11333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:48.786813 11333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:48.786871 11333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:48.786911 11333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:48.792376 11333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:43655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:39793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:39793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:48.793632 11333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:48.794741 11333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:48.803287 11339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:48.803359 11338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:48.803340 11333 server_base.cc:1034] running on GCE node
W20260929 10:27:48.803435 11341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:48.804159 11333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:48.805320 11333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:48.806478 11333 hybrid_clock.cc:648] HybridClock initialized: now 1790677668806445 us; error 46 us; skew 500 ppm
Sep 29 10:27:48 dist-test-slave-2rbl krb5kdc[11320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677668, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:48.809358 11333 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:48.810220 11333 webserver.cc:460] Webserver started at http://127.21.33.190:36129/ using document root <none> and password file <none>
I20260929 10:27:48.810774 11333 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:48.810837 11333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:48.811076 11333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:48.813094 11333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "734fdba0ba7f442586b24d6a045ae9f7"
format_stamp: "Formatted at 2026-09-29 10:27:48 on dist-test-slave-2rbl"
server_key: "a800b5446b8988284d091a61b2cf9916"
server_key_iv: "7dbcbf98fc40bb721ffd4bee307c60e7"
server_key_version: "encryptionkey@0"
I20260929 10:27:48.813740 11333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "734fdba0ba7f442586b24d6a045ae9f7"
format_stamp: "Formatted at 2026-09-29 10:27:48 on dist-test-slave-2rbl"
server_key: "a800b5446b8988284d091a61b2cf9916"
server_key_iv: "7dbcbf98fc40bb721ffd4bee307c60e7"
server_key_version: "encryptionkey@0"
I20260929 10:27:48.817451 11333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:27:48.819839 11348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:48.821069 11333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:48.821282 11333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "734fdba0ba7f442586b24d6a045ae9f7"
format_stamp: "Formatted at 2026-09-29 10:27:48 on dist-test-slave-2rbl"
server_key: "a800b5446b8988284d091a61b2cf9916"
server_key_iv: "7dbcbf98fc40bb721ffd4bee307c60e7"
server_key_version: "encryptionkey@0"
I20260929 10:27:48.821400 11333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:48.826037 11333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:48.829383 11333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:48.829581 11333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:48.836524 11333 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:39793
I20260929 10:27:48.836580 11400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:39793 every 8 connection(s)
I20260929 10:27:48.837555 11333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:48.840312 11401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:48.845736 11401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7: Bootstrap starting.
I20260929 10:27:48.846905 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11333
I20260929 10:27:48.847043 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:48.847325 21638 external_mini_cluster.cc:1428] Setting key 822a9f6e41a3a2026723304b98e5b33c
I20260929 10:27:48.848599 11401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:48.849395 11401 log.cc:824] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:48.851617 11401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7: No bootstrap required, opened a new log
I20260929 10:27:48.852380 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:48.848446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49701 (local address 127.21.33.190:39793)
0929 10:27:48.848906 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:48.848923 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:48.848967 (+ 44us) server_negotiation.cc:407] Connection header received
0929 10:27:48.849721 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:48.849739 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:48.850015 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:48.850303 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:48.851115 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:48.851120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:48.851131 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:48.851141 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:48.851192 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:48.851227 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:48.851311 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:48.851455 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 10:27:48.851629 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":159,"threads_started":1}
I20260929 10:27:48.855259 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:48.855682 11401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fdba0ba7f442586b24d6a045ae9f7" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39793 } }
I20260929 10:27:48.855919 11401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 10:27:48.855996 11401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734fdba0ba7f442586b24d6a045ae9f7, State: Initialized, Role: FOLLOWER
I20260929 10:27:48.856418 11401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fdba0ba7f442586b24d6a045ae9f7" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39793 } }
I20260929 10:27:48.856565 11401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 10:27:48.856657 11401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 10:27:48.856777 11401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:48.857971 11401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fdba0ba7f442586b24d6a045ae9f7" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39793 } }
I20260929 10:27:48.858368 11401 leader_election.cc:302] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 734fdba0ba7f442586b24d6a045ae9f7; no voters:
I20260929 10:27:48.858757 11401 leader_election.cc:288] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 10:27:48.858822 11411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:48.859026 11411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [term 1 LEADER]: Becoming Leader. State: Replica: 734fdba0ba7f442586b24d6a045ae9f7, State: Running, Role: LEADER
I20260929 10:27:48.859261 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:48.857415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33747 (local address 127.21.33.190:39793)
0929 10:27:48.857549 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:48.857554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:48.857772 (+ 218us) server_negotiation.cc:407] Connection header received
0929 10:27:48.857903 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:48.857905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:48.857935 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:48.857998 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:48.858673 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:48.858676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:48.858678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 10:27:48.858688 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:48.858727 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 10:27:48.858737 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:48.858839 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:48.859090 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 10:27:48.859137 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:27:48.859355 11411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fdba0ba7f442586b24d6a045ae9f7" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39793 } }
I20260929 10:27:48.860092 11401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:48.861613 11413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "734fdba0ba7f442586b24d6a045ae9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fdba0ba7f442586b24d6a045ae9f7" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39793 } } }
I20260929 10:27:48.861745 11413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:48.862129 11420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:48.862532 11412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 734fdba0ba7f442586b24d6a045ae9f7. Latest consensus state: current_term: 1 leader_uuid: "734fdba0ba7f442586b24d6a045ae9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734fdba0ba7f442586b24d6a045ae9f7" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 39793 } } }
I20260929 10:27:48.862641 11412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:48.864885 11420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:48.870498 11420 catalog_manager.cc:1347] Generated new cluster ID: 4fd0450c759e44bca427b62033f0cf85
I20260929 10:27:48.870630 11420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:48.884589 11420 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:48.884922 11420 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:48.890642 11420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 734fdba0ba7f442586b24d6a045ae9f7: Generated new TSK 0
I20260929 10:27:48.891353 11420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 10:27:48.896071 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11333
2026-09-29T10:27:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1393 ms)
[----------] 18 tests from AuthTokenIssuingTest (25515 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:48 dist-test-slave-2rbl krb5kdc[11432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:48 dist-test-slave-2rbl krb5kdc[11432](info): set up 2 sockets
Sep 29 10:27:48 dist-test-slave-2rbl krb5kdc[11432](info): commencing operation
krb5kdc: starting...
W20260929 10:27:49.942162 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:50.068584 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33805
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:33805,127.21.33.189:43739,127.21.33.188:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:50.179085 11445 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:27:50.179382 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:50.179407 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:50.179446 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:50.184044 11445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:50.184105 11445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:50.184125 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:50.184142 11445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:50.184157 11445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:50.189546 11445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:33805,127.21.33.189:43739,127.21.33.188:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:33805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:50.190583 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:50.191569 11445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:50.200358 11450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:50.200403 11445 server_base.cc:1034] running on GCE node
W20260929 10:27:50.200327 11453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:50.200344 11451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:50.201122 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:50.202309 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:50.203518 11445 hybrid_clock.cc:648] HybridClock initialized: now 1790677670203502 us; error 26 us; skew 500 ppm
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:50.206300 11445 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:50.207149 11445 webserver.cc:460] Webserver started at http://127.21.33.190:42749/ using document root <none> and password file <none>
I20260929 10:27:50.207672 11445 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:50.207731 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:50.207892 11445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:50.209934 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "cbf1b58224d44a47b7d99ce702026701"
server_key_iv: "508f6d8aa68065f6fe95a242b69a26f7"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.210443 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "cbf1b58224d44a47b7d99ce702026701"
server_key_iv: "508f6d8aa68065f6fe95a242b69a26f7"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.214160 11445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 10:27:50.216796 11460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:50.218168 11445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:50.218310 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "cbf1b58224d44a47b7d99ce702026701"
server_key_iv: "508f6d8aa68065f6fe95a242b69a26f7"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.218425 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:50.246223 11445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:50.249190 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:50.249430 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:50.256649 11445 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:33805
I20260929 10:27:50.256963 11512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:33805 every 8 connection(s)
I20260929 10:27:50.257707 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:50.260625 11513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:50.264452 11513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:50.265128 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11445
I20260929 10:27:50.265296 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:50.265591 21638 external_mini_cluster.cc:1428] Setting key e1db9fa80efe606d9df3b6cd28284d2b
WARNING: no policy specified for kudu/127.21.33.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.189@KRBTEST.COM" created.
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:50.283524 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.266957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33805 (local address 127.0.0.1:35456)
0929 10:27:50.267651 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.267707 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.268883 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.269713 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.269718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.270121 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.270443 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.270458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.272293 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.272295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.273086 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.273096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.274093 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.274097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.274199 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.274884 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.274902 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.278305 (+ 3403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.280389 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.280402 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.280416 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.280765 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.281085 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.281091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.281096 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.281295 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.281993 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.282001 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.282190 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.282406 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.282871 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":88,"spinlock_wait_cycles":22784,"thread_start_us":164,"threads_started":1}
I20260929 10:27:50.283532 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.266988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.21.33.190:33805)
0929 10:27:50.267559 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.267577 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.267852 (+ 275us) server_negotiation.cc:407] Connection header received
0929 10:27:50.269243 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.269252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.269498 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.269697 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.270639 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.272156 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.273246 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.273902 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.274098 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.278473 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.278483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.278501 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.278535 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.280256 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.280881 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.280885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.280886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.280984 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.281400 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.281403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.281404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.281853 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.281956 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.282490 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.282751 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":444,"thread_start_us":155,"threads_started":1}
I20260929 10:27:50.286182 11513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:50.286981 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.286684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37688)
0929 10:27:50.286833 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.286913 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 10:27:50.287166 11463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
I20260929 10:27:50.288064 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.287879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37690)
0929 10:27:50.287973 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.288016 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:27:50.288260 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:43739: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab.
Entry for principal kudu/127.21.33.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab.
I20260929 10:27:50.296972 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:43739
--webserver_interface=127.21.33.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab
--principal=kudu/127.21.33.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:33805,127.21.33.189:43739,127.21.33.188:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:27:50.337527 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } attempt: 1
I20260929 10:27:50.338469 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.338161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37696)
0929 10:27:50.338332 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.338394 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 10:27:50.339198 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.338978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37710)
0929 10:27:50.339097 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.339144 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 10:27:50.339323 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:43739: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
W20260929 10:27:50.407280 11520 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260929 10:27:50.407497 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } attempt: 2
W20260929 10:27:50.407725 11520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:50.407800 11520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:50.407912 11520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:27:50.408363 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.408077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37722)
0929 10:27:50.408219 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.408288 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 10:27:50.409087 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.408876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37726)
0929 10:27:50.408982 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.409034 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:27:50.409260 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:43739: Network error: Client connection negotiation failed: client connection to 127.21.33.189:43739: connect: Connection refused (error 111)
W20260929 10:27:50.412230 11520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:50.412293 11520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:50.412317 11520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:50.412366 11520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:50.412408 11520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:50.418084 11520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:33805,127.21.33.189:43739,127.21.33.188:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab
--principal=kudu/127.21.33.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.33.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:50.419258 11520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:50.420351 11520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:50.429356 11526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:50.429510 11520 server_base.cc:1034] running on GCE node
W20260929 10:27:50.429339 11528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:50.429677 11525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:50.430267 11520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:50.431382 11520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:50.432580 11520 hybrid_clock.cc:648] HybridClock initialized: now 1790677670432544 us; error 48 us; skew 500 ppm
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:50.435631 11520 init.cc:377] Logged in from keytab as kudu/127.21.33.189@KRBTEST.COM (short username kudu)
I20260929 10:27:50.436623 11520 webserver.cc:460] Webserver started at http://127.21.33.189:39649/ using document root <none> and password file <none>
I20260929 10:27:50.437177 11520 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:50.437309 11520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:50.437494 11520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:50.439541 11520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data/instance:
uuid: "4844db457e144d378435a6641db4a935"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "9e9df789d052f05aa04ff8b1f1f04463"
server_key_iv: "34efed0b23cdbfda70c8313a819a40e7"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.440076 11520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/wal/instance:
uuid: "4844db457e144d378435a6641db4a935"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "9e9df789d052f05aa04ff8b1f1f04463"
server_key_iv: "34efed0b23cdbfda70c8313a819a40e7"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.443892 11520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 10:27:50.446257 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:50.447530 11520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:50.447662 11520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/wal
uuid: "4844db457e144d378435a6641db4a935"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "9e9df789d052f05aa04ff8b1f1f04463"
server_key_iv: "34efed0b23cdbfda70c8313a819a40e7"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.447777 11520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:50.486234 11520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:50.489727 11520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:50.489948 11520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:50.497413 11520 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.189:43739
I20260929 10:27:50.497727 11587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.189:43739 every 8 connection(s)
I20260929 10:27:50.498407 11520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
I20260929 10:27:50.500422 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } attempt: 3
I20260929 10:27:50.501426 11588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:50.503414 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11520
I20260929 10:27:50.503562 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-1/wal/instance
I20260929 10:27:50.503903 21638 external_mini_cluster.cc:1428] Setting key b4b7dda3fa78da708a65d29bdbda6e49
I20260929 10:27:50.505792 11588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:50.514433 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.500955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37736)
0929 10:27:50.501106 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.501122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.501266 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.503788 (+ 2522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.503791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.503995 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.504140 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.504150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.506062 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.506067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.506700 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.506711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.507447 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.507451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.507551 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.508107 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.508122 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.509723 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.512306 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.512309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.512311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.512590 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.512943 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.512949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.512952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.513049 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.513871 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.513874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.513987 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.514104 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.514240 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.21.33.188@KRBTEST.COM; defaulting to no policy
I20260929 10:27:50.516206 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.501707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.21.33.189:43739)
0929 10:27:50.502121 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.502139 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.502181 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:50.503036 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.503059 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.503454 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.503821 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.504341 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.505930 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.506835 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.507312 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.507494 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.509871 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.509877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.509885 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.509910 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.512195 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.512710 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.512716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.512722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.512840 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.513169 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.513173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.513174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.513737 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.513897 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.514755 (+ 858us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.515136 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.188@KRBTEST.COM" created.
I20260929 10:27:50.518278 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.507841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.21.33.190:33805)
0929 10:27:50.507989 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.507996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.508448 (+ 452us) server_negotiation.cc:407] Connection header received
0929 10:27:50.508652 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.508658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.508714 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.508837 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.509719 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.510246 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.511309 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.512021 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.512143 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.514774 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.514778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.514782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.514813 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.516311 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.516780 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.516786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.516788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.516870 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.517226 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.517231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.517233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.517407 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.517514 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.518002 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.518131 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:27:50.518324 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.507825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33805 (local address 127.0.0.1:35468)
0929 10:27:50.508323 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.508375 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.508531 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.508915 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.508921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.509407 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.509584 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.509592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.510392 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.510398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.511186 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.511196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.512146 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.512149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.512284 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.513041 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.513056 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.514648 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.516433 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.516442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.516447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.516675 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.516992 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.516995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.516997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.517082 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.517513 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.517519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.517701 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.517915 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.518120 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":156,"threads_started":1}
I20260929 10:27:50.518374 11513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:50.519281 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.519047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53404)
0929 10:27:50.519157 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.519217 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:27:50.519740 11588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:50.519987 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.519722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53420)
0929 10:27:50.519885 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.519933 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 10:27:50.520118 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:46797: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:27:50.527972 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.520488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.21.33.189:43739)
0929 10:27:50.520589 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.520592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.520602 (+ 10us) server_negotiation.cc:407] Connection header received
0929 10:27:50.520636 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.520638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.520666 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.520750 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.521117 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.521863 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.522757 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.522993 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.523098 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.525002 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.525006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.525008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.525030 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.526433 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.526867 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.526873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.526875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.526948 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.527198 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.527201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.527204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.527363 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.527465 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.527719 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.527845 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":46}
I20260929 10:27:50.527972 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.520268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37752)
0929 10:27:50.520370 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.520382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.520475 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.520761 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.520764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.520926 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.521009 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.521014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.521964 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.521970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.522627 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.522636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.523095 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.523098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.523227 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.523705 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.523722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.524872 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.526555 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.526559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.526561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.526772 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.527036 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.527041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.527043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.527126 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.527465 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.527468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.527547 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.527659 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.527795 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":21}
I20260929 10:27:50.528730 11588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:50.529480 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.529149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53436)
0929 10:27:50.529321 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.529415 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260929 10:27:50.530121 11537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
I20260929 10:27:50.531010 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.530795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53442)
0929 10:27:50.530888 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.530931 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:27:50.531270 11588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:46797: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab.
Entry for principal kudu/127.21.33.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab.
I20260929 10:27:50.534183 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:46797
--webserver_interface=127.21.33.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab
--principal=kudu/127.21.33.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:33805,127.21.33.189:43739,127.21.33.188:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:27:50.551265 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } attempt: 1
I20260929 10:27:50.552209 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.551873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53458)
0929 10:27:50.552064 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.552120 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 10:27:50.553013 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.552802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53466)
0929 10:27:50.552909 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.552960 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 10:27:50.553217 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:46797: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
I20260929 10:27:50.580581 11588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } attempt: 1
I20260929 10:27:50.581579 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.581274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53472)
0929 10:27:50.581437 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.581497 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 10:27:50.582273 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.582072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53476)
0929 10:27:50.582168 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.582222 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:27:50.582417 11588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:46797: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
I20260929 10:27:50.628497 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } attempt: 2
I20260929 10:27:50.629372 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.629021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53490)
0929 10:27:50.629175 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.629295 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 10:27:50.630082 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.629871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53492)
0929 10:27:50.629975 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.630022 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 10:27:50.630198 11513 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:46797: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
W20260929 10:27:50.649765 11595 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:27:50.650096 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:50.650158 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:50.650238 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:27:50.650585 11588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } attempt: 2
I20260929 10:27:50.651410 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.651097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53498)
0929 10:27:50.651269 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.651332 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 10:27:50.652132 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.651900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53500)
0929 10:27:50.651999 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.652053 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:27:50.652325 11588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:46797: Network error: Client connection negotiation failed: client connection to 127.21.33.188:46797: connect: Connection refused (error 111)
W20260929 10:27:50.654909 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:50.654983 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:50.655030 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:50.655088 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:50.655126 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:50.660624 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:33805,127.21.33.189:43739,127.21.33.188:46797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab
--principal=kudu/127.21.33.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.33.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:50.661818 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:50.662843 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:50.671283 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:50.671280 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:50.671267 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:50.671645 11595 server_base.cc:1034] running on GCE node
I20260929 10:27:50.672216 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:50.673324 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:50.674489 11595 hybrid_clock.cc:648] HybridClock initialized: now 1790677670674458 us; error 42 us; skew 500 ppm
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:50.677287 11595 init.cc:377] Logged in from keytab as kudu/127.21.33.188@KRBTEST.COM (short username kudu)
I20260929 10:27:50.678156 11595 webserver.cc:460] Webserver started at http://127.21.33.188:44229/ using document root <none> and password file <none>
I20260929 10:27:50.678740 11595 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:50.678805 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:50.679028 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:50.681092 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data/instance:
uuid: "78f72e07c5e143f5acf5e216c01cd2be"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "711ef2a3084b9a93c26b615d3228cabd"
server_key_iv: "f2e7c9dfe03d6d7edecd1dfc5c6cbb99"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.681735 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/wal/instance:
uuid: "78f72e07c5e143f5acf5e216c01cd2be"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "711ef2a3084b9a93c26b615d3228cabd"
server_key_iv: "f2e7c9dfe03d6d7edecd1dfc5c6cbb99"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.685494 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 10:27:50.687880 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:50.689105 11595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:50.689301 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/wal
uuid: "78f72e07c5e143f5acf5e216c01cd2be"
format_stamp: "Formatted at 2026-09-29 10:27:50 on dist-test-slave-2rbl"
server_key: "711ef2a3084b9a93c26b615d3228cabd"
server_key_iv: "f2e7c9dfe03d6d7edecd1dfc5c6cbb99"
server_key_version: "encryptionkey@0"
I20260929 10:27:50.689421 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:50.710245 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:50.713411 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:50.713630 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:50.720917 11595 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.188:46797
I20260929 10:27:50.721236 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.188:46797 every 8 connection(s)
I20260929 10:27:50.722112 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
I20260929 10:27:50.724944 11663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:50.728585 11663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:50.729333 11513 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } attempt: 3
I20260929 10:27:50.730599 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11595
I20260929 10:27:50.730738 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790677387835684-21638-0/minicluster-data/master-2/wal/instance
I20260929 10:27:50.731093 21638 external_mini_cluster.cc:1428] Setting key 5b34d8892261b0b9e8414b771802e097
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:27:50.743333 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.729913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53506)
0929 10:27:50.730071 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.730089 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.730182 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.732622 (+ 2440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.732625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.732827 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.732930 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.732938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.734775 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.734778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.735495 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.735505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.736455 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.736460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.736565 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.737060 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.737072 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.738760 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.741098 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.741104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.741107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.741498 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.741867 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.741871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.741873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.741958 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.742788 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.742792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.742905 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.743057 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.743201 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 10:27:50.743480 11588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } attempt: 3
I20260929 10:27:50.744484 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.730482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.21.33.190:33805)
0929 10:27:50.730614 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.730620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.731365 (+ 745us) server_negotiation.cc:407] Connection header received
0929 10:27:50.732359 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.732362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.732414 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.732483 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.733614 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.734111 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.735330 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.735681 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.735801 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.740325 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.740329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.740331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.740360 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.742142 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.742668 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.742672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.742673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.742758 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.743115 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.743121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.743125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.743359 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.743769 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.744210 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.744338 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 10:27:50.745498 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.730638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.21.33.188:46797)
0929 10:27:50.731177 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.731193 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.731241 (+ 48us) server_negotiation.cc:407] Connection header received
0929 10:27:50.732134 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.732163 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.732427 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.733045 (+ 618us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.733247 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.734654 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.735631 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.736348 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.736496 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.738915 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.738923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.738939 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.738975 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.740951 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.741644 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.741649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.741655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.741763 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.742066 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.742069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.742071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.742674 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.743715 (+ 1041us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.743917 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.744236 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":414,"server-negotiator.queue_time_us":426,"spinlock_wait_cycles":39040,"thread_start_us":156,"threads_started":1}
I20260929 10:27:50.745496 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.730638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:33805 (local address 127.0.0.1:35470)
0929 10:27:50.731228 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.731284 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.732148 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.732542 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.732551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.733044 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.733466 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.733484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.734219 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.734224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.735201 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.735211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.735843 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.735847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.736119 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.737331 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.737356 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.740161 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.742256 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.742263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.742271 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.742547 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.742868 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.742871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.742872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.742955 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.743832 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.743842 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.743974 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.744659 (+ 685us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.744857 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":78,"thread_start_us":149,"threads_started":1}
I20260929 10:27:50.745494 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.731654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.21.33.188:46797)
0929 10:27:50.731935 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.731946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.732065 (+ 119us) server_negotiation.cc:407] Connection header received
0929 10:27:50.732189 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.732192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.732427 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.732613 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.733524 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.734654 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.735586 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.736348 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.736514 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.740665 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.740671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.740676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.740737 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.743340 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.743860 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.743864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.743865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.743959 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.744290 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.744296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.744299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.744505 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.744682 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.745035 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.745148 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":243,"server-negotiator.queue_time_us":218,"spinlock_wait_cycles":22016,"thread_start_us":96,"threads_started":1}
I20260929 10:27:50.746834 11663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:27:50.751972 11513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: Bootstrap starting.
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:27:50.755023 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.743998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53524)
0929 10:27:50.744258 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.744272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.744349 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.746115 (+ 1766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.746118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.746304 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.746394 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.746400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.747343 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.747347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.748173 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.748188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.748858 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.748862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.748971 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.749491 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.749505 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.751027 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.753216 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.753220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.753222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.753474 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.753865 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.753868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.753870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.753964 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.754428 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.754431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.754530 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.754647 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.754815 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:50.755596 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.744479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.21.33.188:46797)
0929 10:27:50.745866 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.745869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.745882 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:27:50.745914 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.745917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.745957 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.746494 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.746558 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.747222 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.748301 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.748728 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.748842 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.751219 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.751223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.751226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.751256 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.753067 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.753597 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.753600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.753602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.753759 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.754092 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.754096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.754098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.754307 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.754434 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.755270 (+ 836us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.755388 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":1329,"thread_start_us":118,"threads_started":1}
I20260929 10:27:50.755883 11513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:50.757777 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.747517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:43739 (local address 127.0.0.1:37768)
0929 10:27:50.747740 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.747754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.747838 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.748062 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.748065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.748242 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.748416 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.748424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.750264 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.750267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.751204 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.751212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.751999 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.752004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.752114 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.752518 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.752534 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.753992 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.756195 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.756200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.756202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.756446 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.756751 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.756754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.756756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.756840 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.757274 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.757277 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.757379 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.757487 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.757622 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":26}
I20260929 10:27:50.757792 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.747589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.21.33.189:43739)
0929 10:27:50.747693 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.747708 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.747835 (+ 127us) server_negotiation.cc:407] Connection header received
0929 10:27:50.747914 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.747916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.747960 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.748029 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.749358 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.750132 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.751318 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.751665 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.751740 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.754127 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.754134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.754139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.754161 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.756099 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.756565 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.756571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.756573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.756656 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.756936 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.756939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.756941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.757115 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.757288 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.757555 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.757661 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 10:27:50.759255 11588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: Bootstrap starting.
I20260929 10:27:50.760195 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.749054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.21.33.190:33805)
0929 10:27:50.749161 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.749165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.749378 (+ 213us) server_negotiation.cc:407] Connection header received
0929 10:27:50.749560 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.749564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.749605 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.749651 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.750516 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.751192 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.752441 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.752779 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.752853 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.756709 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.756714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.756716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.756749 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.758489 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.759001 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.759004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.759005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.759113 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.759452 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.759456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.759457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.759632 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.759770 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.759979 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.760089 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 10:27:50.761688 11663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:50.762023 11588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:50.762229 11513 log.cc:824] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:50.762930 11588 log.cc:824] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:50.765074 11588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: No bootstrap required, opened a new log
I20260929 10:27:50.765090 11513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: No bootstrap required, opened a new log
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
Sep 29 10:27:50 dist-test-slave-2rbl krb5kdc[11432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:27:50.769613 11588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.769620 11513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.769876 11588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4844db457e144d378435a6641db4a935, State: Initialized, Role: FOLLOWER
I20260929 10:27:50.769876 11513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57cb0d9e2d1f41fbb2e5f9aaefed7ef0, State: Initialized, Role: FOLLOWER
I20260929 10:27:50.770221 11588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.770219 11513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.771694 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.762153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:46797 (local address 127.0.0.1:53534)
0929 10:27:50.762503 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:50.762515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:50.762597 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:50.762832 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:50.762834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:50.762996 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:50.763093 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.763101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.763996 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.763999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.764694 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:50.764704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.765344 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.765347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:50.765435 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.765941 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:50.765955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:50.768106 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:50.770156 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.770159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.770161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.770518 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.770793 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:50.770795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:50.770797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:50.770866 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:50.771250 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:50.771252 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:50.771332 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 10:27:50.771409 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:27:50.771538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308}
I20260929 10:27:50.771695 11588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:50.771711 11513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:50.772420 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.761425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.21.33.189:43739)
0929 10:27:50.761533 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.761537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.761614 (+ 77us) server_negotiation.cc:407] Connection header received
0929 10:27:50.761784 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.761788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.761837 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.761891 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.762783 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.763518 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.764585 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.764881 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.764957 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.768314 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.768318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.768321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.768347 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.770479 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.770989 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.770992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.770994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.771077 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.771409 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.771413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.771414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.771618 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.771766 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.772139 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.772270 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 10:27:50.772806 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.762350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.21.33.188:46797)
0929 10:27:50.762445 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.762450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.762672 (+ 222us) server_negotiation.cc:407] Connection header received
0929 10:27:50.762699 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.762701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.762742 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.762820 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.763187 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.763887 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.764821 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.765229 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.765635 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.768243 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.768246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.768249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.768269 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.770040 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.770630 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.770632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.770634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.770710 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.770948 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.770950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.770951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.771153 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.772213 (+ 1060us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.772285 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.772383 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 10:27:50.776057 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:50.776024 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: Bootstrap starting.
I20260929 10:27:50.777312 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.777446 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:50.777642 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.777763 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 10:27:50.777777 11695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:50.777856 11695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:50.778894 11487 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35478
I20260929 10:27:50.779134 11663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:50.779119 11487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 10:27:50.779237 11487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:50.780098 11663 log.cc:824] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:50.780620 11487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.783154 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: No bootstrap required, opened a new log
I20260929 10:27:50.783156 11487 leader_election.cc:288] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [CANDIDATE]: Term 1 election: Requested vote from peers 4844db457e144d378435a6641db4a935 (127.21.33.189:43739), 78f72e07c5e143f5acf5e216c01cd2be (127.21.33.188:46797)
I20260929 10:27:50.784061 11562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4844db457e144d378435a6641db4a935"
I20260929 10:27:50.784250 11562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:50.785534 11562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 in term 1.
I20260929 10:27:50.786135 11463 leader_election.cc:302] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4844db457e144d378435a6641db4a935, 57cb0d9e2d1f41fbb2e5f9aaefed7ef0; no voters:
I20260929 10:27:50.786422 11673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:50.786643 11673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [term 1 LEADER]: Becoming Leader. State: Replica: 57cb0d9e2d1f41fbb2e5f9aaefed7ef0, State: Running, Role: LEADER
I20260929 10:27:50.786980 11673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.787864 11663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.788162 11663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f72e07c5e143f5acf5e216c01cd2be, State: Initialized, Role: FOLLOWER
I20260929 10:27:50.788522 11663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } }
I20260929 10:27:50.785591 11445 master_runner.cc:186] Error getting master registration for 127.21.33.188:46797: OK, instance_id { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" instance_seqno: 1790677670718875 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:27:50.789012 11445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:27:50.779942 11520 master_runner.cc:186] Error getting master registration for 127.21.33.188:46797: OK, instance_id { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" instance_seqno: 1790677670718875 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:27:50.789623 11520 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 10:27:50.790097 11663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:50.790171 11637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "78f72e07c5e143f5acf5e216c01cd2be"
I20260929 10:27:50.790398 11637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:50.790439 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cb0d9e2d1f41fbb2e5f9aaefed7ef0. Latest consensus state: current_term: 1 leader_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.790596 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:50.792392 11637 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 in term 1.
W20260929 10:27:50.793705 11693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:50.793867 11693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 10:27:50.796041 11720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:50.796175 11720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:50.797621 11707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:50.798484 11707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:50.803356 11637 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [term 1 FOLLOWER]: Refusing update from remote peer 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:50.803460 11562 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [term 1 FOLLOWER]: Refusing update from remote peer 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:50.804142 11673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:50.804891 11673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:50.805234 11704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.805348 11704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:50.808138 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cb0d9e2d1f41fbb2e5f9aaefed7ef0. Latest consensus state: current_term: 1 leader_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.808267 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:50.808404 11704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57cb0d9e2d1f41fbb2e5f9aaefed7ef0. Latest consensus state: current_term: 1 leader_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.808537 11704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:50.815683 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.790781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.21.33.189:43739)
0929 10:27:50.790889 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.790893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.801794 (+ 10901us) server_negotiation.cc:407] Connection header received
0929 10:27:50.801963 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.801966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.802016 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.802066 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.802922 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.803643 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.806334 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.806652 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.806732 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.811001 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.811007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.811013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.811045 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.812915 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.813486 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.813492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.813494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.813886 (+ 392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.814532 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.814537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.814541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.814805 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.814941 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.815327 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.815467 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 10:27:50.817016 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.791516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.21.33.188:46797)
0929 10:27:50.791618 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.791622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.804994 (+ 13372us) server_negotiation.cc:407] Connection header received
0929 10:27:50.805167 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.805173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.805276 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.805332 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.807471 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.808172 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.809308 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.809670 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.809772 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.813343 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.813348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.813350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.813380 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.815127 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.815641 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.815644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.815646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.815734 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.816045 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.816051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.816055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.816312 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.816421 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.816758 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.816873 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 10:27:50.821949 11702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.822113 11702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:50.822410 11722 mvcc.cc:205] Tried to move back new op lower bound from 7334615739604172800 to 7334615739553861632. Current Snapshot: MvccSnapshot[applied={T|T < 7334615739604172800}]
I20260929 10:27:50.822535 11704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.822669 11704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:50.822834 11724 mvcc.cc:205] Tried to move back new op lower bound from 7334615739604172800 to 7334615739553861632. Current Snapshot: MvccSnapshot[applied={T|T < 7334615739604172800}]
I20260929 10:27:50.823257 11702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.823367 11702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:50.824522 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57cb0d9e2d1f41fbb2e5f9aaefed7ef0" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 33805 } } peers { permanent_uuid: "4844db457e144d378435a6641db4a935" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 43739 } } peers { permanent_uuid: "78f72e07c5e143f5acf5e216c01cd2be" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 46797 } } }
I20260929 10:27:50.824659 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:50.825069 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:50.790064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.21.33.190:33805)
0929 10:27:50.790192 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.790210 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.801546 (+ 11336us) server_negotiation.cc:407] Connection header received
0929 10:27:50.801731 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.801736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.801796 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.801851 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.803730 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.804500 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.809729 (+ 5229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:50.810132 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:50.810227 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:50.818419 (+ 8192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:50.818423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:50.818426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:50.818459 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:50.820300 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.823593 (+ 3293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.823600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.823602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.823717 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:50.824044 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:50.824048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:50.824050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:50.824278 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:50.824407 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:50.824799 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 10:27:50.824921 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":42}
I20260929 10:27:50.825130 11723 mvcc.cc:205] Tried to move back new op lower bound from 7334615739604172800 to 7334615739553861632. Current Snapshot: MvccSnapshot[applied={T|T < 7334615739604172800}]
I20260929 10:27:50.837245 11707 catalog_manager.cc:1347] Generated new cluster ID: 51b402934f0f4b7fad7e771025bac0cb
I20260929 10:27:50.837342 11707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:50.867075 11707 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:50.868449 11707 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:50.885725 11707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57cb0d9e2d1f41fbb2e5f9aaefed7ef0: Generated new TSK 0
I20260929 10:27:50.886660 11707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:27:50.922987 11516 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.919362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.21.33.190:33805)
0929 10:27:50.919492 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.919497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.921372 (+ 1875us) server_negotiation.cc:407] Connection header received
0929 10:27:50.921721 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.921725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.921777 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.921831 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.922841 (+ 1010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35488: BlockingRecv error: recv got EOF from 127.0.0.1:35488 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 10:27:50.924437 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.920110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.21.33.189:43739)
0929 10:27:50.920221 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.920225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.922570 (+ 2345us) server_negotiation.cc:407] Connection header received
0929 10:27:50.923485 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.923488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.923541 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.923592 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.924319 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37790: BlockingRecv error: recv got EOF from 127.0.0.1:37790 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 10:27:50.925563 11666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.920788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.21.33.188:46797)
0929 10:27:50.920897 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.920902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.923842 (+ 2940us) server_negotiation.cc:407] Connection header received
0929 10:27:50.924804 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.924808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.924861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.924912 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.925442 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53550: BlockingRecv error: recv got EOF from 127.0.0.1:53550 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 10:27:50.929903 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.928154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.21.33.189:43739)
0929 10:27:50.928255 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.928270 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.928744 (+ 474us) server_negotiation.cc:407] Connection header received
0929 10:27:50.928957 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.928961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.929033 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.929084 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.929803 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37792: BlockingRecv error: recv got EOF from 127.0.0.1:37792 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 10:27:50.935135 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.933546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.21.33.189:43739)
0929 10:27:50.933650 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.933654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.934199 (+ 545us) server_negotiation.cc:407] Connection header received
0929 10:27:50.934404 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.934408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.934459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.934507 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.935032 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37794: BlockingRecv error: recv got EOF from 127.0.0.1:37794 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 10:27:50.940745 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.939176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.21.33.189:43739)
0929 10:27:50.939279 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.939283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.939812 (+ 529us) server_negotiation.cc:407] Connection header received
0929 10:27:50.940007 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.940010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.940059 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.940110 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.940624 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37798: BlockingRecv error: recv got EOF from 127.0.0.1:37798 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 10:27:50.948568 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.946887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.21.33.189:43739)
0929 10:27:50.947025 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.947029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.947583 (+ 554us) server_negotiation.cc:407] Connection header received
0929 10:27:50.947808 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.947812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.947867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.947920 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.948458 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37810: BlockingRecv error: recv got EOF from 127.0.0.1:37810 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 10:27:50.961166 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.959234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.21.33.189:43739)
0929 10:27:50.959392 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.959399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.960102 (+ 703us) server_negotiation.cc:407] Connection header received
0929 10:27:50.960332 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.960336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.960393 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.960447 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.961031 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37816: BlockingRecv error: recv got EOF from 127.0.0.1:37816 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 10:27:50.982081 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:50.980125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.21.33.189:43739)
0929 10:27:50.980263 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:50.980269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:50.980939 (+ 670us) server_negotiation.cc:407] Connection header received
0929 10:27:50.981164 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:50.981167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:50.981267 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:50.981325 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:50.981953 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37820: BlockingRecv error: recv got EOF from 127.0.0.1:37820 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 10:27:51.017968 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:51.016584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.21.33.189:43739)
0929 10:27:51.016727 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:51.016732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:51.016985 (+ 253us) server_negotiation.cc:407] Connection header received
0929 10:27:51.017167 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:51.017171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:51.017273 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:51.017362 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:51.017836 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37824: BlockingRecv error: recv got EOF from 127.0.0.1:37824 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 10:27:51.085323 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:51.084219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.21.33.189:43739)
0929 10:27:51.084382 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:51.084386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:51.084512 (+ 126us) server_negotiation.cc:407] Connection header received
0929 10:27:51.084632 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:51.084635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:51.084683 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:51.084768 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:51.085167 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37840: BlockingRecv error: recv got EOF from 127.0.0.1:37840 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 10:27:51.216521 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:51.215410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.21.33.189:43739)
0929 10:27:51.215584 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:51.215588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:51.215657 (+ 69us) server_negotiation.cc:407] Connection header received
0929 10:27:51.215788 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:51.215790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:51.215846 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:51.215918 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:51.216391 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37856: BlockingRecv error: recv got EOF from 127.0.0.1:37856 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 10:27:51.475745 11590 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:51.474687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.21.33.189:43739)
0929 10:27:51.474828 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:51.474831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:51.474943 (+ 112us) server_negotiation.cc:407] Connection header received
0929 10:27:51.475081 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:51.475083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:51.475129 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:51.475220 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:51.475607 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37860: BlockingRecv error: recv got EOF from 127.0.0.1:37860 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 10:27:51.779181 11695 catalog_manager.cc:1259] Loaded cluster ID: 51b402934f0f4b7fad7e771025bac0cb
I20260929 10:27:51.779274 11695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: loading cluster ID for follower catalog manager: success
I20260929 10:27:51.780968 11695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: acquiring CA information for follower catalog manager: success
I20260929 10:27:51.782226 11695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4844db457e144d378435a6641db4a935: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:27:51.797547 11720 catalog_manager.cc:1259] Loaded cluster ID: 51b402934f0f4b7fad7e771025bac0cb
I20260929 10:27:51.797638 11720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: loading cluster ID for follower catalog manager: success
I20260929 10:27:51.799350 11720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: acquiring CA information for follower catalog manager: success
I20260929 10:27:51.800524 11720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78f72e07c5e143f5acf5e216c01cd2be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:27:51.994258 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:51.990075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.21.33.189:43739)
0929 10:27:51.990347 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:51.990351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:51.990367 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:27:51.990417 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:51.990420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:51.990614 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:51.990713 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:51.991005 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:51.991604 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:51.992654 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:51.992886 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:51.992966 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:51.993290 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:51.993771 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:51.993906 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:51.993994 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 10:27:51.994065 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260929 10:27:52.001379 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:51.997548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.21.33.188:46797)
0929 10:27:51.997774 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:51.997778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:51.998026 (+ 248us) server_negotiation.cc:407] Connection header received
0929 10:27:51.998140 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:51.998142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:51.998266 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:51.998361 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:51.998541 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:51.999182 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:51.999965 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:52.000204 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:52.000284 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:52.000521 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:52.000932 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:52.001023 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:52.001097 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 10:27:52.001143 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260929 10:27:52.008455 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:52.004443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.21.33.188:46797)
0929 10:27:52.004577 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:52.004581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:52.004796 (+ 215us) server_negotiation.cc:407] Connection header received
0929 10:27:52.004971 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:52.004975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:52.005021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:52.005071 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:52.005509 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:52.006279 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:52.007279 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:52.007542 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:52.007652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:52.007952 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:52.008043 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:52.008126 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:52.008274 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 10:27:52.008331 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 10:27:52.009037 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:52.004888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.21.33.189:43739)
0929 10:27:52.005035 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:52.005039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:52.005438 (+ 399us) server_negotiation.cc:407] Connection header received
0929 10:27:52.005618 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:52.005621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:52.005676 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:52.005763 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:52.005987 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:52.006594 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:52.007788 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:52.008117 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:52.008227 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:52.008477 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:52.008557 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:52.008632 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:52.008818 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 10:27:52.008887 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 10:27:52.009876 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:52.004338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.21.33.190:33805)
0929 10:27:52.004701 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:52.004708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:52.004726 (+ 18us) server_negotiation.cc:407] Connection header received
0929 10:27:52.004836 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:52.004839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:52.004995 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:52.005111 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:52.005421 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:52.006304 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:52.007554 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:52.007917 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:52.008016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:52.008387 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:52.008774 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:52.008853 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:52.009639 (+ 786us) server_negotiation.cc:299] Negotiation successful
0929 10:27:52.009709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":65,"threads_started":1}
I20260929 10:27:52.018992 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11445
I20260929 10:27:52.025244 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11520
I20260929 10:27:52.031095 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11595
2026-09-29T10:27:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3136 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 10:27:52 dist-test-slave-2rbl krb5kdc[11813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 10:27:52 dist-test-slave-2rbl krb5kdc[11813](info): set up 2 sockets
Sep 29 10:27:52 dist-test-slave-2rbl krb5kdc[11813](info): commencing operation
krb5kdc: starting...
W20260929 10:27:53.070641 21638 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/test-user.keytab.
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T10:27:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T10:27:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.33.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
Entry for principal kudu/127.21.33.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab.
I20260929 10:27:53.205698 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:41411
--webserver_interface=127.21.33.190
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:41411,127.21.33.189:39889,127.21.33.188:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 10:27:53.316354 11826 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:27:53.316642 11826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:53.316707 11826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:53.316782 11826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:53.320791 11826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:53.320895 11826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:53.320935 11826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:53.320973 11826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:53.321012 11826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:53.326241 11826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:41411,127.21.33.189:39889,127.21.33.188:42157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.190.keytab
--principal=kudu/127.21.33.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.190:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.33.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:53.327783 11826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:53.328810 11826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:53.337301 11834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:53.337292 11831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:53.337308 11832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:53.337630 11826 server_base.cc:1034] running on GCE node
I20260929 10:27:53.338330 11826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:53.339437 11826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:53.340637 11826 hybrid_clock.cc:648] HybridClock initialized: now 1790677673340625 us; error 35 us; skew 500 ppm
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:53.343451 11826 init.cc:377] Logged in from keytab as kudu/127.21.33.190@KRBTEST.COM (short username kudu)
I20260929 10:27:53.344367 11826 webserver.cc:460] Webserver started at http://127.21.33.190:33519/ using document root <none> and password file <none>
I20260929 10:27:53.344928 11826 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:53.345019 11826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:53.345319 11826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:53.347340 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data/instance:
uuid: "4944d5fea3b141ab9e684032aba83f28"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "343abc923ce1d78502e01342bde0010a"
server_key_iv: "43a87cdfc2c88be5e389e4ccdad3502d"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.347915 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/wal/instance:
uuid: "4944d5fea3b141ab9e684032aba83f28"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "343abc923ce1d78502e01342bde0010a"
server_key_iv: "43a87cdfc2c88be5e389e4ccdad3502d"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.351656 11826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 10:27:53.354148 11841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:53.355443 11826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 10:27:53.355588 11826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/wal
uuid: "4944d5fea3b141ab9e684032aba83f28"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "343abc923ce1d78502e01342bde0010a"
server_key_iv: "43a87cdfc2c88be5e389e4ccdad3502d"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.355710 11826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:53.374622 11826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:53.378154 11826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:53.378374 11826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:53.385923 11826 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.190:41411
I20260929 10:27:53.386169 11893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.190:41411 every 8 connection(s)
I20260929 10:27:53.386991 11826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/data/info.pb
I20260929 10:27:53.390019 11894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:53.392053 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11826
I20260929 10:27:53.392192 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-0/wal/instance
I20260929 10:27:53.392484 21638 external_mini_cluster.cc:1428] Setting key 1e1096b816cbfdaf28ca396897ca2b20
I20260929 10:27:53.393992 11894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.33.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.189@KRBTEST.COM" created.
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:53.411278 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.396512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.21.33.190:41411)
0929 10:27:53.397044 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.397065 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.397400 (+ 335us) server_negotiation.cc:407] Connection header received
0929 10:27:53.398673 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.398681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.398975 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.399190 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.400008 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.401535 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.402599 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.402989 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.403164 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.406089 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.406098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.406115 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.406146 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.408005 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.408511 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.408517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.408519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.408676 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.409095 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.409097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.409098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.409560 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.409669 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.410216 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.410446 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":351,"spinlock_wait_cycles":54144,"thread_start_us":157,"threads_started":1}
I20260929 10:27:53.411281 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.396512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41411 (local address 127.0.0.1:60266)
0929 10:27:53.397221 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.397272 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.398297 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.399233 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.399238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.399578 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.399842 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.399856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.401664 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.401667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.402457 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.402467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.403160 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.403163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.403269 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.403937 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.403955 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.405917 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.408141 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.408150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.408161 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.408395 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.408806 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.408813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.408817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.408987 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.409698 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.409708 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.409913 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.410133 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.410541 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":46,"thread_start_us":158,"threads_started":1}
I20260929 10:27:53.413937 11894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:53.414710 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.414437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59080)
0929 10:27:53.414560 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.414634 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 10:27:53.414917 11844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
I20260929 10:27:53.415841 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.415611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59092)
0929 10:27:53.415735 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.415788 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 10:27:53.416004 11894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:39889: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab.
Entry for principal kudu/127.21.33.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab.
I20260929 10:27:53.424466 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:39889
--webserver_interface=127.21.33.189
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab
--principal=kudu/127.21.33.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:41411,127.21.33.189:39889,127.21.33.188:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 10:27:53.465281 11894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } attempt: 1
I20260929 10:27:53.466117 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.465811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59100)
0929 10:27:53.465972 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.466039 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 10:27:53.466960 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.466735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59116)
0929 10:27:53.466847 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.466899 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 10:27:53.467092 11894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:39889: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
I20260929 10:27:53.535250 11894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } attempt: 2
I20260929 10:27:53.536111 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.535823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59118)
0929 10:27:53.535971 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.536033 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 10:27:53.536202 11901 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:27:53.536617 11901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:53.536684 11901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:53.536761 11901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 10:27:53.536953 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.536679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59124)
0929 10:27:53.536852 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.536904 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 10:27:53.537081 11894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.189:39889: Network error: Client connection negotiation failed: client connection to 127.21.33.189:39889: connect: Connection refused (error 111)
W20260929 10:27:53.541002 11901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:53.541095 11901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:53.541143 11901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:53.541179 11901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:53.541240 11901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:53.546597 11901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:41411,127.21.33.189:39889,127.21.33.188:42157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.189.keytab
--principal=kudu/127.21.33.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.189:39889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.33.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:53.547708 11901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:53.548766 11901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 10:27:53.557353 11909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:53.557489 11901 server_base.cc:1034] running on GCE node
W20260929 10:27:53.557331 11906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:53.557337 11907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:53.558331 11901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:53.559773 11901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:53.561000 11901 hybrid_clock.cc:648] HybridClock initialized: now 1790677673560977 us; error 38 us; skew 500 ppm
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:53.563887 11901 init.cc:377] Logged in from keytab as kudu/127.21.33.189@KRBTEST.COM (short username kudu)
I20260929 10:27:53.564842 11901 webserver.cc:460] Webserver started at http://127.21.33.189:39677/ using document root <none> and password file <none>
I20260929 10:27:53.565539 11901 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:53.565634 11901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:53.565876 11901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:53.567976 11901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data/instance:
uuid: "c91b12ef1da04f85a808dc9ff7817d5c"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "ba0f034c472b3b0c259dd59da02d4259"
server_key_iv: "f2ffba7618c0148c298c80f5a69de2b9"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.568744 11901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/wal/instance:
uuid: "c91b12ef1da04f85a808dc9ff7817d5c"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "ba0f034c472b3b0c259dd59da02d4259"
server_key_iv: "f2ffba7618c0148c298c80f5a69de2b9"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.573513 11901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 10:27:53.575855 11916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:53.577062 11901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 10:27:53.577237 11901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/wal
uuid: "c91b12ef1da04f85a808dc9ff7817d5c"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "ba0f034c472b3b0c259dd59da02d4259"
server_key_iv: "f2ffba7618c0148c298c80f5a69de2b9"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.577344 11901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:53.588193 11901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:53.590999 11901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:53.591203 11901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:53.598289 11901 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.189:39889
I20260929 10:27:53.598582 11968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.189:39889 every 8 connection(s)
I20260929 10:27:53.599323 11901 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/data/info.pb
I20260929 10:27:53.600448 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11901
I20260929 10:27:53.600593 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-1/wal/instance
I20260929 10:27:53.600880 21638 external_mini_cluster.cc:1428] Setting key 902529666d0111260fb7ffb78a076873
I20260929 10:27:53.602363 11969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:53.606276 11969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.33.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.33.188@KRBTEST.COM" created.
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
I20260929 10:27:53.622233 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.608244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.21.33.190:41411)
0929 10:27:53.608434 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.608441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.609338 (+ 897us) server_negotiation.cc:407] Connection header received
0929 10:27:53.610335 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.610342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.610406 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.610487 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.611876 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.612753 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.614624 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.614865 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.614955 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.618139 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.618142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.618144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.618172 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.619700 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.620359 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.620362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.620363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.620436 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.620961 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.620964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.620966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.621274 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.621376 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.621952 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.622087 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 10:27:53.623123 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.608611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41411 (local address 127.0.0.1:60272)
0929 10:27:53.609096 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.609146 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.610106 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.610661 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.610668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.611121 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.611711 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.611725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.612916 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.612922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.614479 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.614495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.615037 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.615044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.615301 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.615970 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.615997 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.617940 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.619824 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.619831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.619849 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.620244 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.620515 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.620517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.620518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.620661 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.621385 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.621390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.621660 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.621821 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.622189 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":123,"threads_started":1}
I20260929 10:27:53.624593 11969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.33.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab.
Entry for principal kudu/127.21.33.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab.
I20260929 10:27:53.628265 11894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } attempt: 3
I20260929 10:27:53.628683 21638 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
/tmp/dist-test-tasksWWfjY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:42157
--webserver_interface=127.21.33.188
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab
--principal=kudu/127.21.33.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.33.148:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.33.190:41411,127.21.33.189:39889,127.21.33.188:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:27:53.636379 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.625125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59140)
0929 10:27:53.625318 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.625334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.625411 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.626108 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.626111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.626268 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.626421 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.626428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.627477 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.627487 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.628127 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.628143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.629151 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.629155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.629276 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.629798 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.629818 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.631491 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.633709 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.633714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.633719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.634011 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.634389 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.634394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.634398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.635297 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.635833 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.635835 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.635916 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.636034 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.636172 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 10:27:53.636646 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.625376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.21.33.189:39889)
0929 10:27:53.625723 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.625732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.625751 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:27:53.625786 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.625790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.625989 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.626099 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.626531 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.627346 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.628287 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.628917 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.629014 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.631642 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.631654 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.631674 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.631714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.633586 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.634145 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.634152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.634157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.634253 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.635421 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.635424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.635426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.635719 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.635873 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.636137 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.636268 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":152,"threads_started":1}
I20260929 10:27:53.639092 11969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:53.639822 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.639559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34110)
0929 10:27:53.639673 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.639760 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 10:27:53.640101 11918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
I20260929 10:27:53.641068 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.640780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34118)
0929 10:27:53.640925 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.640984 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 10:27:53.641288 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.628725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59156)
0929 10:27:53.628907 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.628922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.629004 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.630625 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.630628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.630795 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.630889 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.630895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.631756 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.631763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.633302 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.633316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.633923 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.633926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.634046 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.634543 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.634563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.636445 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.639311 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.639317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.639322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.639614 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.640006 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.640009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.640011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.640123 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.640714 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.640720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.640838 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.640951 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.641103 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 10:27:53.641319 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.628823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.21.33.189:39889)
0929 10:27:53.630389 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.630400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.630416 (+ 16us) server_negotiation.cc:407] Connection header received
0929 10:27:53.630460 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.630463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.630499 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.630602 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.631007 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.631634 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.633435 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.633807 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.633916 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.637439 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.637443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.637446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.637473 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.639168 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.639762 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.639767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.639771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.639867 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.640271 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.640277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.640281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.640539 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.640712 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.641001 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.641103 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":98,"threads_started":1}
W20260929 10:27:53.641458 11969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:42157: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
I20260929 10:27:53.642035 11894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:53.642701 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.642466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34124)
0929 10:27:53.642576 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.642626 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 10:27:53.643404 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.643186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34138)
0929 10:27:53.643306 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.643350 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 10:27:53.643545 11894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:42157: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
I20260929 10:27:53.674723 11894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } attempt: 1
I20260929 10:27:53.675613 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.675309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34140)
0929 10:27:53.675465 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.675531 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 10:27:53.676383 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.676141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34148)
0929 10:27:53.676276 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.676324 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 10:27:53.676535 11894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:42157: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
I20260929 10:27:53.690727 11969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } attempt: 1
I20260929 10:27:53.691622 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.691284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34156)
0929 10:27:53.691455 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.691529 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 10:27:53.692380 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.692140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34166)
0929 10:27:53.692258 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.692320 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 10:27:53.692509 11969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:42157: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
W20260929 10:27:53.744765 11976 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 10:27:53.745090 11976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 10:27:53.745114 11976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 10:27:53.745162 11976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 10:27:53.749270 11976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 10:27:53.749337 11976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 10:27:53.749357 11976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 10:27:53.749374 11976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 10:27:53.749389 11976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 10:27:53.751698 11894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } attempt: 2
I20260929 10:27:53.752581 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.752281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34172)
0929 10:27:53.752426 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.752498 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 10:27:53.753331 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.753088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34182)
0929 10:27:53.753228 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.753280 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 10:27:53.753494 11894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:42157: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
I20260929 10:27:53.755122 11976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.33.148:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.33.190:41411,127.21.33.189:39889,127.21.33.188:42157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/krb5kdc/kudu_127.21.33.188.keytab
--principal=kudu/127.21.33.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.33.188:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.33.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 09:44:11 UTC on ccycloud.quasar-jhrwyf.root.comops.site
build id 4679897
I20260929 10:27:53.756227 11976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 10:27:53.757349 11976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 10:27:53.760663 11969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } attempt: 2
I20260929 10:27:53.761410 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.761083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34190)
0929 10:27:53.761253 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.761317 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 10:27:53.762193 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.761903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34198)
0929 10:27:53.762073 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.762127 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 10:27:53.762305 11969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.33.188:42157: Network error: Client connection negotiation failed: client connection to 127.21.33.188:42157: connect: Connection refused (error 111)
W20260929 10:27:53.766105 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 10:27:53.766139 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:53.766244 11976 server_base.cc:1034] running on GCE node
W20260929 10:27:53.766153 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 10:27:53.766955 11976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 10:27:53.768085 11976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 10:27:53.769270 11976 hybrid_clock.cc:648] HybridClock initialized: now 1790677673769233 us; error 49 us; skew 500 ppm
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 10:27:53.771971 11976 init.cc:377] Logged in from keytab as kudu/127.21.33.188@KRBTEST.COM (short username kudu)
I20260929 10:27:53.772806 11976 webserver.cc:460] Webserver started at http://127.21.33.188:41301/ using document root <none> and password file <none>
I20260929 10:27:53.773392 11976 fs_manager.cc:359] Metadata directory not provided
I20260929 10:27:53.773456 11976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 10:27:53.773624 11976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 10:27:53.775610 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data/instance:
uuid: "5d193dbffc8a4e4f86db6f75e57222b3"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "54839a6e5c3d24955a1070937c3c9e9a"
server_key_iv: "1f0cf6bcd07596da926a913967c32dee"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.776115 11976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/wal/instance:
uuid: "5d193dbffc8a4e4f86db6f75e57222b3"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "54839a6e5c3d24955a1070937c3c9e9a"
server_key_iv: "1f0cf6bcd07596da926a913967c32dee"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.780027 11976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 10:27:53.782609 11992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 10:27:53.783874 11976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 10:27:53.783977 11976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data,/tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/wal
uuid: "5d193dbffc8a4e4f86db6f75e57222b3"
format_stamp: "Formatted at 2026-09-29 10:27:53 on dist-test-slave-2rbl"
server_key: "54839a6e5c3d24955a1070937c3c9e9a"
server_key_iv: "1f0cf6bcd07596da926a913967c32dee"
server_key_version: "encryptionkey@0"
I20260929 10:27:53.784089 11976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 10:27:53.814411 11976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 10:27:53.817878 11976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 10:27:53.818041 11976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 10:27:53.825253 11976 rpc_server.cc:307] RPC server started. Bound to: 127.21.33.188:42157
I20260929 10:27:53.825575 12044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.33.188:42157 every 8 connection(s)
I20260929 10:27:53.826275 11976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/data/info.pb
I20260929 10:27:53.829005 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 10:27:53.832738 12045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:53.836254 21638 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu as pid 11976
I20260929 10:27:53.836407 21638 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksWWfjY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790677387835684-21638-0/minicluster-data/master-2/wal/instance
I20260929 10:27:53.836760 21638 external_mini_cluster.cc:1428] Setting key 7ea9b04476170ebf703a5ab95616b4b0
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:27:53.847659 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.834696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.21.33.190:41411)
0929 10:27:53.834874 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.834879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.835712 (+ 833us) server_negotiation.cc:407] Connection header received
0929 10:27:53.836724 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.836728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.836792 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.836892 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.838501 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.839318 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.840708 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.841035 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.841144 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.843768 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.843772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.843777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.843807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.845521 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.846081 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.846084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.846087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.846170 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.846519 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.846523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.846524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.846726 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.846833 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.847338 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.847462 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 10:27:53.848645 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.837501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.21.33.188:42157)
0929 10:27:53.837871 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.837882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.837907 (+ 25us) server_negotiation.cc:407] Connection header received
0929 10:27:53.837988 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.837993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.838182 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.838351 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.839232 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.840549 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.841625 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.842135 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.842248 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.844781 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.844790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.844808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.844829 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.846548 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.846993 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.847000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.847001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.847123 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.847408 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.847413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.847415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.847652 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.847887 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.848064 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.848205 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":2304,"thread_start_us":178,"threads_started":1}
I20260929 10:27:53.848652 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.835085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.190:41411 (local address 127.0.0.1:60286)
0929 10:27:53.835536 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.835592 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.836484 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.837079 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.837105 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.837682 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.838342 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.838358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.839426 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.839429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.840588 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.840599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.841178 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.841181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.841393 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.841993 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.842014 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.843622 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.845654 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.845668 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.845680 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.845965 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.846264 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.846268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.846270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.846418 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.846832 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.846839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.847103 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.847299 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.847619 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":125,"threads_started":1}
I20260929 10:27:53.849867 12045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:53.852661 11894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } attempt: 3
I20260929 10:27:53.853437 11969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } attempt: 3
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.190@KRBTEST.COM
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.190@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.189@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:27:53.861498 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.850485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.21.33.189:39889)
0929 10:27:53.850641 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.850645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.850665 (+ 20us) server_negotiation.cc:407] Connection header received
0929 10:27:53.850701 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.850703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.850741 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.850827 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.851287 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.851928 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.852916 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.853318 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.853444 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.857522 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.857526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.857528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.857564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.859560 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.860080 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.860086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.860089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.860172 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.860479 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.860482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.860484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.860701 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.860771 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.861157 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.861325 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 10:27:53.862018 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.850368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.189:39889 (local address 127.0.0.1:59168)
0929 10:27:53.850465 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.850479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.850553 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.850839 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.850842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.851036 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.851173 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.851180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.852054 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.852057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.852580 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.852587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.853578 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.853582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.853696 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.855301 (+ 1605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.855313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.857381 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.859721 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.859724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.859728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.859976 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.860281 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.860286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.860290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.860373 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.860850 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.860856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.860962 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.861714 (+ 752us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.861866 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 10:27:53.862938 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.851149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.21.33.190:41411)
0929 10:27:53.851264 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.851268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.851280 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:27:53.851377 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.851380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.851422 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.851507 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.852048 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.852594 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.853736 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.854048 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.854115 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.858193 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.858196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.858199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.858243 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.860267 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.860880 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.860885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.860888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.860972 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.861351 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.861356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.861361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.861636 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.861876 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.862599 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.862731 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":8576}
I20260929 10:27:53.863003 12045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } has no permanent_uuid. Determining permanent_uuid...
I20260929 10:27:53.865010 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.853045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34208)
0929 10:27:53.853229 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.853242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.853311 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.853569 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.853572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.853758 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.853838 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.853843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.854788 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.854793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.855598 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.855610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.856199 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.856204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.856297 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.856756 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.856770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.858273 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.862223 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.862226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.862228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.862475 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.863094 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.863097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.863099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.863726 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.864442 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.864445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.864576 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.864685 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.864820 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"spinlock_wait_cycles":5504}
I20260929 10:27:53.865643 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.853121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.21.33.188:42157)
0929 10:27:53.853290 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.853294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.853306 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:27:53.853398 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.853401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.853438 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.853933 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.854009 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.854683 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.855748 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.856063 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.856518 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.858409 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.858414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.858420 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.858938 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.862095 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.862597 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.862600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.862601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.862956 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.864059 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.864062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.864063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.864304 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.864396 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.865378 (+ 982us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.865485 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 10:27:53.867434 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.855895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34214)
0929 10:27:53.856073 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.856088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.856165 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.856950 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.856953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.857153 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.857283 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.857294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.858332 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.858336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.859226 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.859237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.859896 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.859902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.859989 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.860444 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.860462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.862182 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.865903 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.865909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.865913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.866168 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.866471 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.866474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.866476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.866573 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.866971 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.866974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.867061 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.867138 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.867271 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 10:27:53.868012 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.855554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.21.33.188:42157)
0929 10:27:53.856425 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.856437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.856720 (+ 283us) server_negotiation.cc:407] Connection header received
0929 10:27:53.856758 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.856761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.856806 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.856902 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.857419 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.858196 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.859381 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.859769 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.860217 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.862326 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.862331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.862334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.862359 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.865784 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.866290 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.866293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.866295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.866373 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.866675 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.866678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.866679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.866875 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.867609 (+ 734us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.867683 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.867794 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":120,"threads_started":1}
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.33.188@KRBTEST.COM for kudu/127.21.33.188@KRBTEST.COM
I20260929 10:27:53.870682 11894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28: Bootstrap starting.
Sep 29 10:27:53 dist-test-slave-2rbl krb5kdc[11813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790677673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.33.189@KRBTEST.COM
I20260929 10:27:53.871883 11969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: Bootstrap starting.
I20260929 10:27:53.872696 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.863620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.33.188:42157 (local address 127.0.0.1:34220)
0929 10:27:53.863862 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 10:27:53.863877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 10:27:53.863945 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 10:27:53.864623 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 10:27:53.864626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 10:27:53.864793 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 10:27:53.864881 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.864887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.865948 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.865954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.866777 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 10:27:53.866784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.867341 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.867345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 10:27:53.867420 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.867880 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 10:27:53.867893 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 10:27:53.869358 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 10:27:53.871305 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.871308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.871309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.871530 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.871812 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 10:27:53.871814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 10:27:53.871816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 10:27:53.871891 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 10:27:53.872280 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 10:27:53.872282 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 10:27:53.872354 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 10:27:53.872426 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 10:27:53.872544 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260929 10:27:53.873250 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.863648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.21.33.188:42157)
0929 10:27:53.864402 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.864410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.864422 (+ 12us) server_negotiation.cc:407] Connection header received
0929 10:27:53.864454 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.864457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.864498 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.864588 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.864990 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.865852 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.866889 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.867235 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.867608 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.869485 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.869488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.869490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.869511 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.871194 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.871648 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.871650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.871652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.871725 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.871989 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.871993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.871994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.872182 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.872860 (+ 678us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.872931 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.873036 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":114,"threads_started":1}
I20260929 10:27:53.874733 11894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:53.875564 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.864421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.21.33.189:39889)
0929 10:27:53.864548 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.864552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.864710 (+ 158us) server_negotiation.cc:407] Connection header received
0929 10:27:53.864844 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.864847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.864895 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.865022 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.866350 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.867039 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.868119 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.868424 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.868510 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.871991 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.871995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.871998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.872024 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.873754 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.874249 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.874253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.874255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.874342 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.874683 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.874687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.874688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.874880 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.875050 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.875287 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.875399 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:27:53.875706 11894 log.cc:824] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:53.876295 12045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: Bootstrap starting.
I20260929 10:27:53.877770 11894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28: No bootstrap required, opened a new log
I20260929 10:27:53.878350 11969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:53.878545 12045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: Neither blocks nor log segments found. Creating new log.
I20260929 10:27:53.879387 12045 log.cc:824] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:53.879606 11969 log.cc:824] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: Log is configured to *not* fsync() on all Append() calls
I20260929 10:27:53.881011 21638 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 10:27:53.881973 12045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: No bootstrap required, opened a new log
I20260929 10:27:53.882323 11969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: No bootstrap required, opened a new log
I20260929 10:27:53.882289 11868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4944d5fea3b141ab9e684032aba83f28"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60296
I20260929 10:27:53.883517 11894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.883844 11894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4944d5fea3b141ab9e684032aba83f28, State: Initialized, Role: FOLLOWER
I20260929 10:27:53.884392 11894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.885044 12055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.884999 11969 raft_consensus.cc:348] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.885181 12055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:53.885253 11969 raft_consensus.cc:729] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c91b12ef1da04f85a808dc9ff7817d5c, State: Initialized, Role: FOLLOWER
I20260929 10:27:53.885552 11969 consensus_queue.cc:260] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.886035 12056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.886224 12056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:53.886788 11868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 10:27:53.886943 11868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:53.886839 12045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.887063 12045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d193dbffc8a4e4f86db6f75e57222b3, State: Initialized, Role: FOLLOWER
I20260929 10:27:53.887136 11969 sys_catalog.cc:564] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:53.887355 12045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.887835 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.887979 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:53.888386 11868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.888633 12045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:53.889612 11868 leader_election.cc:288] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [CANDIDATE]: Term 1 election: Requested vote from peers c91b12ef1da04f85a808dc9ff7817d5c (127.21.33.189:39889), 5d193dbffc8a4e4f86db6f75e57222b3 (127.21.33.188:42157)
I20260929 10:27:53.886724 11894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 10:27:53.891040 11943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4944d5fea3b141ab9e684032aba83f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c91b12ef1da04f85a808dc9ff7817d5c"
I20260929 10:27:53.891274 11943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:53.892653 11943 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4944d5fea3b141ab9e684032aba83f28 in term 1.
I20260929 10:27:53.893581 11844 leader_election.cc:302] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4944d5fea3b141ab9e684032aba83f28, c91b12ef1da04f85a808dc9ff7817d5c; no voters:
I20260929 10:27:53.893018 12019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4944d5fea3b141ab9e684032aba83f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d193dbffc8a4e4f86db6f75e57222b3"
I20260929 10:27:53.893828 12055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 10:27:53.893895 12019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 10:27:53.894047 12055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [term 1 LEADER]: Becoming Leader. State: Replica: 4944d5fea3b141ab9e684032aba83f28, State: Running, Role: LEADER
I20260929 10:27:53.894353 12055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } }
I20260929 10:27:53.895291 12019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4944d5fea3b141ab9e684032aba83f28 in term 1.
I20260929 10:27:53.896164 12055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4944d5fea3b141ab9e684032aba83f28. Latest consensus state: current_term: 1 leader_uuid: "4944d5fea3b141ab9e684032aba83f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.896255 12055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: This master's current role is: LEADER
W20260929 10:27:53.896775 12068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:53.896838 12068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 10:27:53.897636 12094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 10:27:53.894174 11901 master_runner.cc:186] Error getting master registration for 127.21.33.190:41411: OK, instance_id { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" instance_seqno: 1790677673383894 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 10:27:53.897786 11901 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 10:27:53.897812 12094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 10:27:53.902285 12080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 10:27:53.902371 12080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 10:27:53.910460 12099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 10:27:53.913684 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.898459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.21.33.189:39889)
0929 10:27:53.898582 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.898588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.900969 (+ 2381us) server_negotiation.cc:407] Connection header received
0929 10:27:53.902232 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.902235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.902288 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.902380 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.903459 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.904209 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.906412 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.906769 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.906880 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.908816 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.908820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.908823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.908848 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.911715 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.912237 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.912241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.912243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.912329 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.912646 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.912650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.912652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.912872 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.912989 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.913378 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.913504 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 10:27:53.916891 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.900018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.21.33.190:41411)
0929 10:27:53.900118 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.900122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.902546 (+ 2424us) server_negotiation.cc:407] Connection header received
0929 10:27:53.903466 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.903470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.903525 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.903570 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.904350 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.905079 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.910197 (+ 5118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.910529 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.910642 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.912704 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.912709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.912712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.912740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.914842 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.915511 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.915515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.915517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.915619 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.915951 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.915954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.915956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.916177 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.916304 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.916600 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.916734 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 10:27:53.912742 12099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 10:27:53.922691 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:53.899125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.21.33.188:42157)
0929 10:27:53.899243 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:53.899248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:53.901165 (+ 1917us) server_negotiation.cc:407] Connection header received
0929 10:27:53.902263 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:53.902267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:53.902322 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:53.902396 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:53.909497 (+ 7101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.910289 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.913111 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:53.913474 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:53.913544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:53.919005 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 10:27:53.919009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 10:27:53.919012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 10:27:53.919043 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 10:27:53.920794 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.921334 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.921337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.921340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.921427 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 10:27:53.921719 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 10:27:53.921723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 10:27:53.921724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 10:27:53.921942 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 10:27:53.922050 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:53.922374 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 10:27:53.922504 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 10:27:53.937918 12019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [term 1 FOLLOWER]: Refusing update from remote peer 4944d5fea3b141ab9e684032aba83f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:53.937919 11943 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [term 1 FOLLOWER]: Refusing update from remote peer 4944d5fea3b141ab9e684032aba83f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 10:27:53.938731 12096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:53.939335 12055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 10:27:53.943208 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4944d5fea3b141ab9e684032aba83f28. Latest consensus state: current_term: 1 leader_uuid: "4944d5fea3b141ab9e684032aba83f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.943378 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:53.943974 12056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4944d5fea3b141ab9e684032aba83f28. Latest consensus state: current_term: 1 leader_uuid: "4944d5fea3b141ab9e684032aba83f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.944092 12056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:53.947527 12055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4944d5fea3b141ab9e684032aba83f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.947680 12055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:53.948166 12104 mvcc.cc:205] Tried to move back new op lower bound from 7334615752441991168 to 7334615752274518016. Current Snapshot: MvccSnapshot[applied={T|T < 7334615752441991168}]
I20260929 10:27:53.949316 12056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4944d5fea3b141ab9e684032aba83f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.949456 12056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:53.953357 12106 mvcc.cc:205] Tried to move back new op lower bound from 7334615752441991168 to 7334615752274518016. Current Snapshot: MvccSnapshot[applied={T|T < 7334615752441991168}]
I20260929 10:27:53.955636 12099 catalog_manager.cc:1347] Generated new cluster ID: 52b9d5be433744b4b0ae041cf53fabd9
I20260929 10:27:53.955729 12099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 10:27:53.956383 12096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4944d5fea3b141ab9e684032aba83f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.956523 12096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28 [sys.catalog]: This master's current role is: LEADER
I20260929 10:27:53.957634 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4944d5fea3b141ab9e684032aba83f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4944d5fea3b141ab9e684032aba83f28" member_type: VOTER last_known_addr { host: "127.21.33.190" port: 41411 } } peers { permanent_uuid: "c91b12ef1da04f85a808dc9ff7817d5c" member_type: VOTER last_known_addr { host: "127.21.33.189" port: 39889 } } peers { permanent_uuid: "5d193dbffc8a4e4f86db6f75e57222b3" member_type: VOTER last_known_addr { host: "127.21.33.188" port: 42157 } } }
I20260929 10:27:53.957775 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 10:27:53.959491 12105 mvcc.cc:205] Tried to move back new op lower bound from 7334615752441991168 to 7334615752274518016. Current Snapshot: MvccSnapshot[applied={T|T < 7334615752441991168}]
I20260929 10:27:53.992213 12099 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 10:27:53.995301 12099 catalog_manager.cc:1504] Loading token signing keys...
I20260929 10:27:54.012936 12099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4944d5fea3b141ab9e684032aba83f28: Generated new TSK 0
I20260929 10:27:54.013967 12099 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 10:27:54.057190 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.054197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.21.33.189:39889)
0929 10:27:54.054366 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.054373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.056087 (+ 1714us) server_negotiation.cc:407] Connection header received
0929 10:27:54.056297 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.056301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.056353 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.056408 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.057045 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59208: BlockingRecv error: recv got EOF from 127.0.0.1:59208 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 10:27:54.058864 12051 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.054895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.21.33.188:42157)
0929 10:27:54.055022 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.055027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.057873 (+ 2846us) server_negotiation.cc:407] Connection header received
0929 10:27:54.057985 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.057989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.058045 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.058143 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.058734 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34238: BlockingRecv error: recv got EOF from 127.0.0.1:34238 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 10:27:54.059190 11898 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.055492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.21.33.190:41411)
0929 10:27:54.055604 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.055608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.058159 (+ 2551us) server_negotiation.cc:407] Connection header received
0929 10:27:54.058332 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.058336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.058391 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.058443 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.059073 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60306: BlockingRecv error: recv got EOF from 127.0.0.1:60306 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 10:27:54.064183 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.062533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.21.33.189:39889)
0929 10:27:54.062649 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.062654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.063186 (+ 532us) server_negotiation.cc:407] Connection header received
0929 10:27:54.063416 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.063420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.063498 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.063552 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.064067 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59210: BlockingRecv error: recv got EOF from 127.0.0.1:59210 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 10:27:54.070529 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.067905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59226 (local address 127.21.33.189:39889)
0929 10:27:54.068011 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.068015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.068561 (+ 546us) server_negotiation.cc:407] Connection header received
0929 10:27:54.068792 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.068796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.068863 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.068923 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.070412 (+ 1489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59226: BlockingRecv error: recv got EOF from 127.0.0.1:59226 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 10:27:54.076766 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.074999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.21.33.189:39889)
0929 10:27:54.075103 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.075119 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.075803 (+ 684us) server_negotiation.cc:407] Connection header received
0929 10:27:54.076012 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.076016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.076066 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.076127 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.076648 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59238: BlockingRecv error: recv got EOF from 127.0.0.1:59238 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 10:27:54.084929 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.083176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.21.33.189:39889)
0929 10:27:54.083325 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.083329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.083902 (+ 573us) server_negotiation.cc:407] Connection header received
0929 10:27:54.084133 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.084136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.084194 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.084252 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.084813 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59240: BlockingRecv error: recv got EOF from 127.0.0.1:59240 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 10:27:54.097519 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.095674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.21.33.189:39889)
0929 10:27:54.095796 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.095801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.096439 (+ 638us) server_negotiation.cc:407] Connection header received
0929 10:27:54.096658 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.096662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.096720 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.096777 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.097400 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59246: BlockingRecv error: recv got EOF from 127.0.0.1:59246 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 10:27:54.117400 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.116052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.21.33.189:39889)
0929 10:27:54.116188 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.116192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.116487 (+ 295us) server_negotiation.cc:407] Connection header received
0929 10:27:54.116595 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.116599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.116665 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.116766 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.117277 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59252: BlockingRecv error: recv got EOF from 127.0.0.1:59252 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 10:27:54.152477 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.151390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.21.33.189:39889)
0929 10:27:54.151534 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.151538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.151687 (+ 149us) server_negotiation.cc:407] Connection header received
0929 10:27:54.151816 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.151818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.151866 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.151950 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.152378 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59254: BlockingRecv error: recv got EOF from 127.0.0.1:59254 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 10:27:54.220013 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.218736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.21.33.189:39889)
0929 10:27:54.218912 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.218917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.219067 (+ 150us) server_negotiation.cc:407] Connection header received
0929 10:27:54.219271 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.219274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.219333 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.219427 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.219893 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59266: BlockingRecv error: recv got EOF from 127.0.0.1:59266 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 10:27:54.351337 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.350190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.21.33.189:39889)
0929 10:27:54.350322 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.350325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.350470 (+ 145us) server_negotiation.cc:407] Connection header received
0929 10:27:54.350563 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.350565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.350614 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.350695 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.351171 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59282: BlockingRecv error: recv got EOF from 127.0.0.1:59282 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 10:27:54.610901 11975 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 10:27:54.609713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.21.33.189:39889)
0929 10:27:54.609850 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:54.609854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:54.609955 (+ 101us) server_negotiation.cc:407] Connection header received
0929 10:27:54.610088 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:54.610091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:54.610142 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:54.610219 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 10:27:54.610758 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59290: BlockingRecv error: recv got EOF from 127.0.0.1:59290 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 10:27:54.898197 12068 catalog_manager.cc:1259] Loaded cluster ID: 52b9d5be433744b4b0ae041cf53fabd9
I20260929 10:27:54.898325 12068 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: loading cluster ID for follower catalog manager: success
I20260929 10:27:54.900027 12068 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: acquiring CA information for follower catalog manager: success
I20260929 10:27:54.901289 12068 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c91b12ef1da04f85a808dc9ff7817d5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:27:54.903468 12080 catalog_manager.cc:1259] Loaded cluster ID: 52b9d5be433744b4b0ae041cf53fabd9
I20260929 10:27:54.903537 12080 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: loading cluster ID for follower catalog manager: success
I20260929 10:27:54.905184 12080 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: acquiring CA information for follower catalog manager: success
I20260929 10:27:54.906363 12080 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d193dbffc8a4e4f86db6f75e57222b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 10:27:55.128535 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:55.124934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.21.33.189:39889)
0929 10:27:55.125185 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:55.125189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:55.125231 (+ 42us) server_negotiation.cc:407] Connection header received
0929 10:27:55.125378 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:55.125380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:55.125506 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:55.125603 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:55.125832 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.126347 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.127189 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.127418 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.127492 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:55.127771 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:55.128197 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:55.128287 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:55.128357 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 10:27:55.128405 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260929 10:27:55.135665 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:55.131699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.21.33.188:42157)
0929 10:27:55.131922 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:55.131926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:55.131939 (+ 13us) server_negotiation.cc:407] Connection header received
0929 10:27:55.132102 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:55.132104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:55.132230 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:55.132327 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:55.132520 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.133023 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.133967 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.134202 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.134285 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:55.134636 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:55.135081 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:55.135186 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:55.135288 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 10:27:55.135344 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260929 10:27:55.142838 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:55.139298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.21.33.188:42157)
0929 10:27:55.139413 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:55.139419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:55.139515 (+ 96us) server_negotiation.cc:407] Connection header received
0929 10:27:55.139702 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:55.139708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:55.139790 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:55.139863 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:55.140146 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.140706 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.141702 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.141981 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.142095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:55.142327 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:55.142410 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:55.142476 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:55.142628 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 10:27:55.142699 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 10:27:55.143278 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:55.139108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.21.33.189:39889)
0929 10:27:55.139246 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:55.139251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:55.139614 (+ 363us) server_negotiation.cc:407] Connection header received
0929 10:27:55.139814 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:55.139820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:55.139883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:55.139950 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:55.140283 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.140815 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.142017 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.142364 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.142474 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:55.142746 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:55.142823 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:55.142870 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:55.143058 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 10:27:55.143115 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 10:27:55.144562 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 10:27:55.139192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.21.33.190:41411)
0929 10:27:55.139562 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 10:27:55.139569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 10:27:55.139588 (+ 19us) server_negotiation.cc:407] Connection header received
0929 10:27:55.139708 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 10:27:55.139714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 10:27:55.139907 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 10:27:55.140030 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 10:27:55.141012 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.141848 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.142905 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 10:27:55.143253 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 10:27:55.143354 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 10:27:55.143689 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 10:27:55.144101 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 10:27:55.144199 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 10:27:55.144310 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 10:27:55.144387 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":73,"threads_started":1}
I20260929 10:27:55.148206 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11826
I20260929 10:27:55.155483 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11901
I20260929 10:27:55.161432 21638 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksWWfjY/build/debug/bin/kudu with pid 11976
2026-09-29T10:27:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3132 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6269 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287321 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST